Claude 3.5 Haiku vs JT-MINI
Wins 2 of 6 areas
Long documents · Following instructions
Wins 3 of 6 areas
Coding · Agents · Reasoning
JT-MINI wins more areas, narrowly.Claude 3.5 Haiku is better at long documents.
Scores updated · 15 tests both models report · How we compare
Where each one wins
Tests won in each of the six areas where both have results. Each piece is one test, so a longer bar means more evidence; grey means the two scored within a point of each other.
- ReasoningHard problems that need careful thinking02JT-MINI2 of 3 tests · 1 tie
- CodingWriting and fixing software01JT-MINI1 of 2 tests · 1 tie
- AgentsCarrying out multi-step tasks on its own01JT-MINI1 of 1 test
- Long documentsFinding answers in very long texts10Claude 3.5 Haiku1 of 1 test
- Following instructionsDoing exactly what it is asked10Claude 3.5 Haiku1 of 1 test
- FactsGetting facts right instead of making them up11Even1 each
Agents, long documents and following instructions rest on a single test each.
The biggest differences
The three tests each model wins by the widest margin. Scores are out of 100.
Where JT-MINI pulls ahead
- Graduate-level biology, physics and chemistry questionsGPQA Diamond+26.8points ahead
- Hard command-line tasks in a real terminalTerminal-Bench Hard+15.9points ahead
- Very hard expert questions across many subjectsHumanity's Last Exam+2.8points ahead
Where Claude 3.5 Haiku pulls ahead
- Avoids making up answers it doesn't knowAA-Omniscience · Non-hallucination+50.2points ahead
- Reasons across sets of long documentsAA-LCR+14.3points ahead
- Follows unfamiliar, precisely checkable instructionsIFBench+6.1points ahead
Every test, side by side
All 15 tests both models report. The winning score is in its model's colour; marks a score checked independently.
ReasoningJT-MINI
- GPQA DiamondJT-MINI by 26.840.867.6+26.8
- Humanity's Last ExamJT-MINI by 2.83.66.4+2.8
- CritPttie00tie
FactsEven
- AA-Omniscience · Non-hallucinationClaude 3.5 Haiku by 50.258.98.7+50.2
- AA-Omniscience · AccuracyJT-MINI by 113.214.2+1
Following instructionsClaude 3.5 Haiku
- IFBenchClaude 3.5 Haiku by 6.142.836.7+6.1
Other results5 tests, not counted
Tests outside the eight areas. They are not counted above: several are summary scores built from other tests, or the same test under another name.
- τ²-Bench Telecom (AA run)JT-MINI by 68.424.693+68.4
- AA-OmniscienceClaude 3.5 Haiku by 41.6-22.5-64.1+41.6
- AA Agentic IndexJT-MINI by 40.51.942.4+40.5
- Artificial Analysis Coding IndexJT-MINI by 5.315.921.2+5.3
- AA IntelligenceJT-MINI by 3.38.912.2+3.3
Questions people ask
Which is better, Claude 3.5 Haiku or JT-MINI?
JT-MINI wins three of the six areas where both have results: coding, agents and reasoning. Claude 3.5 Haiku wins long documents and following instructions. They are level on facts.
Which is better for coding?
JT-MINI. It wins 1 of the 2 coding tests both models report; Claude 3.5 Haiku wins none, and 1 is a tie.
How do you compare the two?
We use the 15 benchmark tests both models have published scores on. The verdict counts the 10 tests in the eight capability areas, and a gap under one point (ten on rating-style scales) counts as a tie. The other 5 are listed but not counted, because several are summary scores or repeat a test. Each score is the one shown on the model's own page.