Claude 3 Haiku vs GPT-4o mini
Wins 1 of 5 areas
Following instructions
Wins 3 of 5 areas
Coding · Reasoning · Images and charts
GPT-4o mini is the stronger all-rounder.Claude 3 Haiku is better at following instructions.
Scores updated · 25 tests both models report · How we compare
Where each one wins
Tests won in each of the five areas where both have results. Each piece is one test, so a longer bar means more evidence; grey means the two scored within a point of each other.
- ReasoningHard problems that need careful thinking01GPT-4o mini1 of 2 tests · 1 tie
- CodingWriting and fixing software01GPT-4o mini1 of 1 test
- Images and chartsUnderstanding pictures, charts and video01GPT-4o mini1 of 1 test
- Following instructionsDoing exactly what it is asked10Claude 3 Haiku1 of 1 test
- AgentsCarrying out multi-step tasks on its own00Even0 each · 1 tie
Coding, agents, images and charts and following instructions rest on a single test each.
What it costs
Prices per million tokens, roughly 750,000 words. The bars show the cost of a million tokens read plus a million written.
GPT-4o mini costs 50% less for the same work.
The biggest differences
The tests each model wins by the widest margin, up to three each. Scores are out of 100.
Where GPT-4o mini pulls ahead
- Harder college exam questions with imagesMMMU-Pro+10.7points ahead
- Graduate-level biology, physics and chemistry questionsGPQA Diamond+5.2points ahead
- Code for real scientific research problemsSciCode+4.3points ahead
Where Claude 3 Haiku pulls ahead
- Follows unfamiliar, precisely checkable instructionsIFBench+5.1points ahead
Every test, side by side
All 25 tests both models report. The winning score is in its model's colour; marks a score checked independently.
ReasoningGPT-4o mini
- GPQA DiamondGPT-4o mini by 5.237.442.6+5.2
- Humanity's Last Examtie4.14.2tie
Following instructionsClaude 3 Haiku
- IFBenchClaude 3 Haiku by 5.136.131+5.1
Other results19 tests, not counted
Tests outside the eight areas. They are not counted above: several are summary scores built from other tests, or the same test under another name.
- NarrativeQAGPT-4o mini by 52.424.476.8+52.4
- AIR-Bench 2024Claude 3 Haiku by 26.482.756.3+26.4
- naturalquestions_closedbookGPT-4o mini by 24.214.438.5+24.2
- livecodebench_easyGPT-4o mini by 20.661.181.7+20.6
- livecodebench_mediumGPT-4o mini by 16.2925.2+16.2
- LiveCodeBenchGPT-4o mini by 1322.535.5+13
- MGSMGPT-4o mini by 11.975.187+11.9
- HumanEvalGPT-4o mini by 11.375.987.2+11.3
- XSTestGPT-4o mini by 10.785.396+10.7
- OpenBookQAGPT-4o mini by 8.283.892+8.2
- HarmBenchClaude 3 Haiku by 6.491.384.9+6.4
- Artificial Analysis Coding IndexGPT-4o mini by 4.76.711.4+4.7
- GSM8KClaude 3 Haiku by 4.688.984.3+4.6
- livecodebench_hardGPT-4o mini by 2.41.94.3+2.4
- simple_safety_testsClaude 3 Haiku by 2.210097.8+2.2
- anthropic_red_teamClaude 3 Haiku by 1.710098.3+1.7
- DROPGPT-4o mini by 1.378.479.7+1.3
- AA IntelligenceGPT-4o mini by 1.15.66.7+1.1
- MMLUtie66.266.8tie
Questions people ask
Which is better, Claude 3 Haiku or GPT-4o mini?
GPT-4o mini wins three of the five areas where both have results: coding, reasoning and images and charts. Claude 3 Haiku wins following instructions. They are level on agents.
Which is better for coding?
GPT-4o mini. It wins the one coding test both models report.
Which is cheaper?
Claude 3 Haiku costs $0.25 per million input tokens and $1.25 per million output tokens; GPT-4o mini costs $0.15 and $0.60. That makes GPT-4o mini about 50% cheaper for the same work.
How do you compare the two?
We use the 25 benchmark tests both models have published scores on. The verdict counts the 6 tests in the eight capability areas, and a gap under one point (ten on rating-style scales) counts as a tie. The other 19 are listed but not counted, because several are summary scores or repeat a test. Each score is the one shown on the model's own page.