Claude 3.5 Sonnet vs Claude 3 Opus
Wins 3 of 3 areas
Coding · Reasoning · Images and charts
Wins 0 of 3 areas
—
Claude 3.5 Sonnet is the stronger all-rounder.
Scores updated · 41 tests both models report · How we compare
Where each one wins
Tests won in each of the three areas where both have results. Each piece is one test, so a longer bar means more evidence; grey means the two scored within a point of each other.
Coding and images and charts rest on a single test each.
What it costs
Prices per million tokens, roughly 750,000 words. The bars show the cost of a million tokens read plus a million written.
Claude 3.5 Sonnet costs 80% less for the same work.
The biggest differences
The tests each model wins by the widest margin, up to three each. Scores are out of 100.
Where Claude 3.5 Sonnet pulls ahead
- Math problems shown in pictures and chartsMathVista+17.2points ahead
- Code for real scientific research problemsSciCode+8.3points ahead
- Graduate-level biology, physics and chemistry questionsGPQA Diamond+7.1points ahead
Where Claude 3 Opus pulls ahead
No clear win on a test scored out of 100.
Every test, side by side
All 41 tests both models report. The winning score is in its model's colour; marks a score checked independently.
ReasoningClaude 3.5 Sonnet
- GPQA DiamondClaude 3.5 Sonnet by 7.15648.9+7.1
- SimpleBenchClaude 3.5 Sonnet by 427.523.5+4
- Humanity's Last Examtie3.22.8tie
Images and chartsClaude 3.5 Sonnet
- MathVistaClaude 3.5 Sonnet by 17.267.750.5+17.2
Other results36 tests, not counted
Tests outside the eight areas. They are not counted above: several are summary scores built from other tests, or the same test under another name.
- NarrativeQAClaude 3.5 Sonnet by 39.574.635.1+39.5
- metr_hcastClaude 3.5 Sonnet by 20.140.720.6+20.1
- MATHClaude 3.5 Sonnet by 1171.160.1+11
- MMLU-ProClaude 3.5 Sonnet by 9.177.668.5+9.1
- GPQA (Diamond) 0-shot CoTClaude 3.5 Sonnet by 959.450.4+9
- MMMU (validation)Claude 3.5 Sonnet by 8.968.359.4+8.9
- HumanEvalClaude 3.5 Sonnet by 8.893.784.9+8.8
- GPQA (Diamond) Maj@32 5-shot CoTClaude 3.5 Sonnet by 7.767.259.5+7.7
- HumanEval 0-shotClaude 3.5 Sonnet by 7.19284.9+7.1
- AI2DClaude 3.5 Sonnet by 6.694.788.1+6.6
- Artificial Analysis Coding IndexClaude 3.5 Sonnet by 6.52619.5+6.5
- BBHClaude 3.5 Sonnet by 6.393.186.8+6.3
- BIG-Bench Hard 3-shot CoTClaude 3.5 Sonnet by 6.393.186.8+6.3
- naturalquestions_closedbookClaude 3.5 Sonnet by 6.150.244+6.1
- DocVQA (test, ANLS score)Claude 3.5 Sonnet by 5.995.289.3+5.9
- BBQ Ambig Accuracy (%)Claude 3 Opus by 593.698.6+5
- metr_re_benchClaude 3.5 Sonnet by 550+5
- BBQ Disambig Bias (%)Claude 3 Opus by 4.5-3.70.8+4.5
- DROPClaude 3.5 Sonnet by 487.183.1+4
- XSTestClaude 3.5 Sonnet by 3.195.692.5+3.1
- BBQ Disambig Accuracy (%)Claude 3 Opus by 2.876.279+2.8
- MMLU 0-shot CoTClaude 3.5 Sonnet by 2.688.385.7+2.6
- MMLU 5-shot CoTClaude 3.5 Sonnet by 2.290.488.2+2.2
- MMLUClaude 3.5 Sonnet by 1.988.786.8+1.9
- OpenBookQAClaude 3.5 Sonnet by 1.697.295.6+1.6
- AA IntelligenceClaude 3 Opus by 1.57.28.7+1.5
- AIR-Bench 2024Claude 3.5 Sonnet by 1.585.984.4+1.5
- GSM8KClaude 3.5 Sonnet by 1.496.495+1.4
- bbqtie94.994tie
- MGSMtie91.690.7tie
- MGSM 0-shot CoTtie91.690.7tie
- HarmBenchtie98.197.4tie
- BBQ Ambig Bias (%)tie0.91.2tie
- metr_swaatie98.798.5tie
- anthropic_red_teamtie99.899.8tie
- simple_safety_teststie100100tie
Questions people ask
Which is better, Claude 3.5 Sonnet or Claude 3 Opus?
Claude 3.5 Sonnet wins all three areas where both have results: coding, reasoning and images and charts. Claude 3 Opus wins none.
Which is better for coding?
Claude 3.5 Sonnet. It wins the one coding test both models report.
Which is cheaper?
Claude 3.5 Sonnet costs $3.00 per million input tokens and $15.00 per million output tokens; Claude 3 Opus costs $15.00 and $75.00. That makes Claude 3.5 Sonnet about 80% cheaper for the same work.
How do you compare the two?
We use the 41 benchmark tests both models have published scores on. The verdict counts the 5 tests in the eight capability areas, and a gap under one point (ten on rating-style scales) counts as a tie. The other 36 are listed but not counted, because several are summary scores or repeat a test. Each score is the one shown on the model's own page.