Claude 3.5 Sonnet vs Gemini 1.5 Pro

Anthropic · released Jun 21, 2024

Wins 1 of 3 areas

Coding

Google · released Sep 24, 2024

Wins 2 of 3 areas

Reasoning · Images and charts

Gemini 1.5 Pro is the stronger all-rounder.Claude 3.5 Sonnet is better at coding.

Scores updated Sep 11, 2026 · 66 tests both models report · How we compare

Where each one wins

Tests won in each of the three areas where both have results. Each piece is one test, so a longer bar means more evidence; grey means the two scored within a point of each other.

ReasoningHard problems that need careful thinking02Gemini 1.5 Pro2 of 3 tests · 1 tie
Images and chartsUnderstanding pictures, charts and video12Gemini 1.5 Pro2 of 5 tests · 2 ties
CodingWriting and fixing software20Claude 3.5 Sonnet2 of 2 tests

What it costs

Prices per million tokens, roughly 750,000 words. The bars show the cost of a million tokens read plus a million written.

Claude 3.5 Sonnet$3.00 to read · $15.00 to write$18Price from Artificial Analysis

Gemini 1.5 ProNo current price is tracked.

The biggest differences

The three tests each model wins by the widest margin. Scores are out of 100.

Where Gemini 1.5 Pro pulls ahead

Visual perception tasks people solve at a glanceBLINK+4.5points aheadGemini 1.5 Pro61Claude 3.5 Sonnet56.5
Graduate-level biology, physics and chemistry questionsGPQA Diamond+2.9points aheadGemini 1.5 Pro58.9Claude 3.5 Sonnet56
Very hard expert questions across many subjectsHumanity's Last Exam+1.4points aheadGemini 1.5 Pro4.6Claude 3.5 Sonnet3.2

Where Claude 3.5 Sonnet pulls ahead

Fixes real GitHub issues in Python projectsSWE-bench Verified+14.8points aheadClaude 3.5 Sonnet49Gemini 1.5 Pro34.2
College exam questions with charts, maps and diagramsMMMU+2.4points aheadClaude 3.5 Sonnet68.3Gemini 1.5 Pro65.9
Code for real scientific research problemsSciCode+2.1points aheadClaude 3.5 Sonnet31.6Gemini 1.5 Pro29.5

Every test, side by side

All 66 tests both models report. The winning score is in its model's colour; marks a score checked independently.

CodingClaude 3.5 Sonnet

SWE-bench VerifiedClaude 3.5 Sonnet by 14.84934.2+14.8
SciCodeClaude 3.5 Sonnet by 2.131.629.5+2.1

Full coding ranking

ReasoningGemini 1.5 Pro

GPQA DiamondGemini 1.5 Pro by 2.95658.9+2.9
Humanity's Last ExamGemini 1.5 Pro by 1.43.24.6+1.4
SimpleBenchtie27.527.1tie

Full reasoning ranking

Images and chartsGemini 1.5 Pro

BLINKGemini 1.5 Pro by 4.556.561+4.5
MMMUClaude 3.5 Sonnet by 2.468.365.9+2.4
OCRBenchGemini 1.5 Pro by 17980+1
MathVistatie67.768.1tie
MMMU-Protie54.755tie

Full images and charts ranking

Other results56 tests, not counted

Tests outside the eight areas. They are not counted above: several are summary scores built from other tests, or the same test under another name.

AIR-Bench 2024Claude 3.5 Sonnet by 18.685.967.3+18.6
HarmBenchClaude 3.5 Sonnet by 18.298.179.9+18.2
MMMU (val) (Pass@1)Claude 3.5 Sonnet by 14.268.354.1+14.2
POPE (test)Gemini 1.5 Pro by 12.776.689.3+12.7
InterGPS (test)Gemini 1.5 Pro by 12.645.658.2+12.6
DROPClaude 3.5 Sonnet by 12.287.174.9+12.2
ScienceQA (img-test)Gemini 1.5 Pro by 12.273.886+12.2
MATHGemini 1.5 Pro by 11.471.182.5+11.4
HumanEvalClaude 3.5 Sonnet by 9.693.784.1+9.6
MMLUClaude 3.5 Sonnet by 9.288.779.5+9.2
HumanEval 0-shotClaude 3.5 Sonnet by 7.99284.1+7.9
Video-MME OverallGemini 1.5 Pro by 6.755.962.6+6.7
Arena HardGemini 1.5 Pro by 6.179.285.3+6.1
MMMU (validation)Claude 3.5 Sonnet by 6.168.362.2+6.1
TextVQA-valClaude 3.5 Sonnet by 670.564.5+6
GSM8KClaude 3.5 Sonnet by 5.696.490.8+5.6
MMBench (dev en)Gemini 1.5 Pro by 5.682.387.9+5.6
MEGA-Bench_macroClaude 3.5 Sonnet by 5.551.445.9+5.5
M-LongDoc (multimodal long-document benchmark)Claude 3.5 Sonnet by 5.231.426.2+5.2
SimpleQAClaude 3.5 Sonnet by 528.423.4+5
naturalquestions_closedbookClaude 3.5 Sonnet by 4.650.245.5+4.6
MGSMClaude 3.5 Sonnet by 4.191.687.5+4.1
Chinese SimpleQA (C-SimpleQA)Gemini 1.5 Pro by 455.459.4+4
BBHClaude 3.5 Sonnet by 3.993.189.2+3.9
BIG-Bench Hard 3-shot CoTClaude 3.5 Sonnet by 3.993.189.2+3.9
DocVQAClaude 3.5 Sonnet by 3.795.291.5+3.7
OlympiadBenchGemini 1.5 Pro by 3.728.432.1+3.7
MTOB eng → kalam (ChrF) no contextClaude 3.5 Sonnet by 3.420.216.8+3.4
XSTestGemini 1.5 Pro by 3.295.698.8+3.2
IFEvalGemini 1.5 Pro by 2.986.589.4+2.9
simple_safety_testsClaude 3.5 Sonnet by 2.510097.5+2.5
Artificial Analysis Coding IndexClaude 3.5 Sonnet by 2.42623.6+2.4
LiveCodeBenchClaude 3.5 Sonnet by 2.332.830.5+2.3
MTOB eng → kalam (ChrF) full bookGemini 1.5 Pro by 2.255.657.9+2.2
ChartQAClaude 3.5 Sonnet by 2.190.888.7+2.1
ChartQA_relaxedClaude 3.5 Sonnet by 2.190.888.7+2.1
DocVQA (test, ANLS score)Claude 3.5 Sonnet by 2.195.293.1+2.1
RULER 128KClaude 3.5 Sonnet by 2.193.891.7+2.1
OpenBookQAClaude 3.5 Sonnet by 297.295.2+2
MMLU-ProClaude 3.5 Sonnet by 1.877.675.8+1.8
MTOB kalam → eng (BLEURT) half bookGemini 1.5 Pro by 1.859.761.5+1.8
RULER 64KClaude 3.5 Sonnet by 1.495.293.8+1.4
NarrativeQAGemini 1.5 Pro by 174.675.6+1
MTOB kalam → eng (BLEURT) full booktie62.363.1tie
Ruler 32ktie9595.8tie
AA Intelligencetie7.27.9tie
IFEval (avg)tie90.189.4tie
MTOB kalam → eng (BLEURT) no contexttie31.432tie
DROP (F1)tie88.889.2tie
AI2Dtie94.794.4tie
MBPP+ (EvalPlus-augmented)tie75.175.4tie
Ruler 16ktie95.796tie
Ruler 4ktie96.596.2tie
anthropic_red_teamtie99.899.9tie
MTOB eng → kalam (ChrF) half booktie53.653.7tie
Ruler 8ktie9696tie

Questions people ask

Which is better, Claude 3.5 Sonnet or Gemini 1.5 Pro?

Gemini 1.5 Pro wins two of the three areas where both have results: reasoning and images and charts. Claude 3.5 Sonnet wins coding.

Which is better for coding?

Claude 3.5 Sonnet. It wins 2 of the 2 coding tests both models report; Gemini 1.5 Pro wins none.

How do you compare the two?

We use the 66 benchmark tests both models have published scores on. The verdict counts the 10 tests in the eight capability areas, and a gap under one point (ten on rating-style scales) counts as a tie. The other 56 are listed but not counted, because several are summary scores or repeat a test. Each score is the one shown on the model's own page.