Which is better, Gemini 1.5 Pro or GPT-4o?

Across 59 shared benchmarks tracked on Vector Wire, Gemini 1.5 Pro leads on 24 and GPT-4o leads on 34 — GPT-4o leads on more. Strengths differ by axis; the table below shows every head-to-head number with provenance.

SEEDRefresh Models —Deals —Regulatory —Sources —

Gemini 1.5 Pro vs GPT-4o

GooglevsOpenAI59 shared benchmarks24–34 head-to-head

Benchmark	Gemini 1.5 Pro	GPT-4o
AA Intelligence	10	11.2
AI2D	80.9	94.2
aider_polyglot	16.9	30.7
AIR-Bench 2024	82.8	62.4
anthropic_red_team	99.9	99.1
ARC-AGI-2	0.8	0
Arena Hard	85.3	92.4
arena_vision	1179	1162
Artificial Analysis Coding Index	23.6	24.2
bbq	94.5	95.1
chartqa	88.7	88.1
ChartQA_relaxed	88.7	88.1
Chinese SimpleQA (C-SimpleQA)	59.4	64.6
docvqa	91.5	92.8
DROP	74.9	83.4
DROP (F1)	89.2	89.2
Fortress	53.9	47.2
GPQA Diamond	59.1	70.1
GSM8K	95.2	95.6
harmbench	79.9	82.9
HLE	4.9	5.3
humaneval	86.6	90.6
ifeval	89.4	84.3
IFEval (avg)	89.4	84.1
LiveCodeBench	30.5	38.3
M-LongDoc (multimodal long-document benchmark)	26.2	41.4
MATH	92	85.3
MathVista	70.6	63.8
MBPP+ (EvalPlus-augmented)	75.4	76.2
MEGA-Bench_macro	45.9	49.4
mgsm	87.5	90.5
mmlu	86.8	88.1
MMLU-Pro	75.8	74.7
MMMU	68.4	72.2
MMMU (val) (Pass@1)	54.1	69.1
MMMU-Pro	55	59.9
MTOB eng → kalam (ChrF) half book	53.7	54.3
MTOB eng → kalam (ChrF) no context	16.8	9.9
MTOB kalam → eng (BLEURT) half book	61.5	58.3
MTOB kalam → eng (BLEURT) no context	32	33.2
narrativeqa	78.3	80.4
naturalquestions_closedbook	45.5	50.1
OCRBench	800	806
OlympiadBench	32.1	25.2
OpenBookQA	95.2	96.8
Ruler 16k	96	89
Ruler 32k	95.8	88.8
Ruler 4k	96.2	97
RULER 64K	93.8	88.4
Ruler 8k	96	92.1
scicode	29.5	33.4
SEAL VISTA	37.1	34.9
simple_safety_tests	97.5	98.5
simplebench	27.1	17.8
simpleqa	23.4	39.4
SWE-bench Verified	34.2	38.8
Video-MME	78.6	77.2
Video-MME (wo subs)	75	71.9
xstest	98.8	97.3

Best tracked score per model per benchmark (default configuration; source-attributed). ↓ marks lower-is-better metrics. Open either model for its full surface, provenance and pricing.