Gemini 2.5 Flash vs Llama 3.1 Instruct 405B
Wins 5 of 6 areas
Coding · Agents · Reasoning · Long documents · Following instructions
Wins 0 of 6 areas
—
Gemini 2.5 Flash is the stronger all-rounder.
Scores updated · 26 tests both models report · How we compare
Where each one wins
Tests won in each of the six areas where both have results. Each piece is one test, so a longer bar means more evidence; grey means the two scored within a point of each other.
- ReasoningHard problems that need careful thinking40Gemini 2.5 Flash4 of 4 tests
- CodingWriting and fixing software20Gemini 2.5 Flash2 of 3 tests · 1 tie
- AgentsCarrying out multi-step tasks on its own10Gemini 2.5 Flash1 of 1 test
- Long documentsFinding answers in very long texts10Gemini 2.5 Flash1 of 1 test
- Following instructionsDoing exactly what it is asked10Gemini 2.5 Flash1 of 1 test
- FactsGetting facts right instead of making them up11Even1 each
Agents, long documents and following instructions rest on a single test each.
What it costs
Prices per million tokens, roughly 750,000 words. The bars show the cost of a million tokens read plus a million written.
The biggest differences
The tests each model wins by the widest margin, up to three each. Scores are out of 100.
Where Gemini 2.5 Flash pulls ahead
- Reasons across sets of long documentsAA-LCR+40points ahead
- Fixes real GitHub issues in Python projectsSWE-bench Verified+35.9points ahead
- Graduate-level biology, physics and chemistry questionsGPQA Diamond+27.5points ahead
Where Llama 3.1 Instruct 405B pulls ahead
- Avoids making up answers it doesn't knowAA-Omniscience · Non-hallucination+22.9points ahead
Every test, side by side
All 26 tests both models report. The winning score is in its model's colour; marks a score checked independently.
CodingGemini 2.5 Flash
- SWE-bench VerifiedGemini 2.5 Flash by 35.960.424.5+35.9
- Terminal-Bench HardGemini 2.5 Flash by 6.813.66.8+6.8
- SciCodetie29.129.9tie
ReasoningGemini 2.5 Flash
- GPQA DiamondGemini 2.5 Flash by 27.57951.5+27.5
- SimpleBenchGemini 2.5 Flash by 18.241.223+18.2
- Humanity's Last ExamGemini 2.5 Flash by 8.112.14+8.1
- CritPtGemini 2.5 Flash by 1.11.10+1.1
FactsEven
- AA-Omniscience · Non-hallucinationLlama 3.1 Instruct 405B by 22.924.747.6+22.9
- AA-Omniscience · AccuracyGemini 2.5 Flash by 2.825.923.2+2.8
Following instructionsGemini 2.5 Flash
- IFBenchGemini 2.5 Flash by 11.350.339+11.3
Other results14 tests, not counted
Tests outside the eight areas. They are not counted above: several are summary scores built from other tests, or the same test under another name.
- AIME 2024Gemini 2.5 Flash by 5982.323.3+59
- LiveCodeBenchGemini 2.5 Flash by 48.576.227.7+48.5
- AA-OmniscienceLlama 3.1 Instruct 405B by 12.7-29.8-17.1+12.7
- τ²-Bench Telecom (AA run)Gemini 2.5 Flash by 12.631.619+12.6
- AA Agentic IndexGemini 2.5 Flash by 12.518.86.3+12.5
- AIR-Bench 2024Gemini 2.5 Flash by 10.168.758.6+10.1
- SimpleQAGemini 2.5 Flash by 9.826.917.1+9.8
- Artificial Analysis Coding IndexGemini 2.5 Flash by 7.722.214.5+7.7
- AA IntelligenceGemini 2.5 Flash by 5.813.17.3+5.8
- bbqGemini 2.5 Flash by 3.297.794.5+3.2
- XSTestGemini 2.5 Flash by 2.998.895.9+2.9
- anthropic_red_teamGemini 2.5 Flash by 2.398.896.5+2.3
- simple_safety_teststie9898.8tie
- HarmBenchtie62.662.7tie
Questions people ask
Which is better, Gemini 2.5 Flash or Llama 3.1 Instruct 405B?
Gemini 2.5 Flash wins five of the six areas where both have results: coding, agents, reasoning, long documents and following instructions. Llama 3.1 Instruct 405B wins none. They are level on facts.
Which is better for coding?
Gemini 2.5 Flash. It wins 2 of the 3 coding tests both models report; Llama 3.1 Instruct 405B wins none, and 1 is a tie.
How do you compare the two?
We use the 26 benchmark tests both models have published scores on. The verdict counts the 12 tests in the eight capability areas, and a gap under one point (ten on rating-style scales) counts as a tie. The other 14 are listed but not counted, because several are summary scores or repeat a test. Each score is the one shown on the model's own page.