Which is better, Claude 3.5 Sonnet or Gemini 1.5 Pro?

Across 67 shared benchmarks tracked on Vector Wire, Claude 3.5 Sonnet leads on 38 and Gemini 1.5 Pro leads on 27 — Claude 3.5 Sonnet leads on more. Strengths differ by axis; the table below shows every head-to-head number with provenance.

SEEDRefresh Models —Deals —Regulatory —Sources —

Claude 3.5 Sonnet vs Gemini 1.5 Pro

AnthropicvsGoogle67 shared benchmarks38–27 head-to-head

Benchmark	Claude 3.5 Sonnet	Gemini 1.5 Pro
AA Intelligence	10	10
AI2D	94.7	80.9
aider_polyglot	45.3	16.9
AIR-Bench 2024	85.9	82.8
anthropic_red_team	99.8	99.9
Arena Hard	87.6	85.3
arena_vision	1146	1179
Artificial Analysis Coding Index	30.2	23.6
bbh	93.1	89.2
bbq	94.9	94.5
BLINK	56.5	61
chartqa	90.8	88.7
ChartQA_relaxed	90.8	88.7
Chinese SimpleQA (C-SimpleQA)	56.8	59.4
docvqa	95.2	91.5
DROP	87.1	74.9
DROP (F1)	88.8	89.2
Fortress	13	53.9
GPQA Diamond	67.2	59.1
GSM8K	96.9	95.2
harmbench	98.1	79.9
HLE	3.9	4.9
humaneval	93.7	86.6
ifeval	90.1	89.4
IFEval (avg)	90.1	89.4
InterGPS (test)	45.6	58.2
LiveCodeBench	32.8	30.5
M-LongDoc (multimodal long-document benchmark)	31.4	26.2
MATH	81.3	92
MathVista	67.7	70.6
MBPP+ (EvalPlus-augmented)	75.1	75.4
MEGA-Bench_macro	51.4	45.9
mgsm	91.6	87.5
MMBench (dev-en)	82.3	87.9
mmlu	88.3	86.8
MMLU-Pro	78	75.8
MMMU	72	68.4
MMMU (val) (Pass@1)	68.3	54.1
MMMU-Pro	54.7	55
MTOB eng → kalam (ChrF) full book	55.6	57.9
MTOB eng → kalam (ChrF) half book	53.6	53.7
MTOB eng → kalam (ChrF) no context	20.2	16.8
MTOB kalam → eng (BLEURT) full book	62.3	63.1
MTOB kalam → eng (BLEURT) half book	59.7	61.5
MTOB kalam → eng (BLEURT) no context	31.4	32
narrativeqa	74.6	78.3
naturalquestions_closedbook	50.2	45.5
OCRBench	790	800
OlympiadBench	28.4	32.1
OpenBookQA	97.2	95.2
POPE (test)	76.6	89.3
RULER 128K	93.8	91.7
Ruler 16k	95.7	96
Ruler 32k	95	95.8
Ruler 4k	96.5	96.2
RULER 64K	95.2	93.8
Ruler 8k	96	96
scicode	36.6	29.5
ScienceQA (img-test)	73.8	86
SEAL VISTA	38.7	37.1
simple_safety_tests	100	97.5
simplebench	41.4	27.1
simpleqa	28.4	23.4
SWE-bench Verified	50.8	34.2
TextVQA (val)	70.5	64.5
Video-MME Overall	55.9	62.6
xstest	95.6	98.8

Best tracked score per model per benchmark (default configuration; source-attributed). ↓ marks lower-is-better metrics. Open either model for its full surface, provenance and pricing.