The two are evenly matched.Kimi K2 Base is better at coding; Llama 3.1 Instruct 405B at reasoning.
Scores updated · 27 tests both models report · How we compare
Where each one wins
Tests won in each of the two areas where both have results. Each piece is one test, so a longer bar means more evidence; grey means the two scored within a point of each other.
Coding and reasoning rest on a single test each.
The biggest differences
The tests each model wins by the widest margin, up to three each. Scores are out of 100.
Where Kimi K2 Base pulls ahead
- Fixes real GitHub issues in Python projectsSWE-bench Verified+3.7points ahead
Where Llama 3.1 Instruct 405B pulls ahead
- Graduate-level biology, physics and chemistry questionsGPQA Diamond+3.4points ahead
Every test, side by side
All 27 tests both models report. The winning score is in its model's colour; marks a score checked independently.
ReasoningLlama 3.1 Instruct 405B
- GPQA DiamondLlama 3.1 Instruct 405B by 3.448.151.5+3.4
Other results25 tests, not counted
Tests outside the eight areas. They are not counted above: several are summary scores built from other tests, or the same test under another name.
- OpenBookQALlama 3.1 Instruct 405B by 43.250.894+43.2
- Chinese SimpleQA (C-SimpleQA)Kimi K2 Base by 24.274.650.4+24.2
- MBPPKimi K2 Base by 20.68968.4+20.6
- C-EvalKimi K2 Base by 2092.572.5+20
- SimpleQAKimi K2 Base by 18.235.317.1+18.2
- CMMLUKimi K2 Base by 17.290.973.7+17.2
- MultiPL-E HumanEvalLlama 3.1 Instruct 405B by 14.760.575.2+14.7
- HumanEvalLlama 3.1 Instruct 405B by 10.878.289+10.8
- CRUXEval-O (output prediction)Kimi K2 Base by 9.769.659.9+9.7
- CRUXEval-I (input prediction)Kimi K2 Base by 9.56858.5+9.5
- MultiPL-E MBPPLlama 3.1 Instruct 405B by 6.958.865.7+6.9
- MGSMLlama 3.1 Instruct 405B by 6.485.291.6+6.4
- BBHKimi K2 Base by 5.888.782.9+5.8
- HellaSwagKimi K2 Base by 5.494.689.2+5.4
- GSM8KLlama 3.1 Instruct 405B by 4.792.196.8+4.7
- MMLU-ProLlama 3.1 Instruct 405B by 4.169.273.3+4.1
- MMLU-ReduxKimi K2 Base by 490.286.2+4
- MMMLUKimi K2 Base by 3.877.673.8+3.8
- MATHLlama 3.1 Instruct 405B by 3.670.273.8+3.6
- DROPLlama 3.1 Instruct 405B by 1.283.684.8+1.2
- PIQALlama 3.1 Instruct 405B by 184.985.9+1
- MBPP+ (EvalPlus-augmented)tie73.873tie
- MMLUtie87.888.6tie
- ARC-Challengetie96.296.9tie
- WinoGrandetie85.385.2tie
Questions people ask
Which is better, Kimi K2 Base or Llama 3.1 Instruct 405B?
Kimi K2 Base and Llama 3.1 Instruct 405B each win one of the two areas where both have results. Kimi K2 Base wins coding; Llama 3.1 Instruct 405B wins reasoning.
Which is better for coding?
Kimi K2 Base. It wins the one coding test both models report.
How do you compare the two?
We use the 27 benchmark tests both models have published scores on. The verdict counts the 2 tests in the eight capability areas, and a gap under one point (ten on rating-style scales) counts as a tie. The other 25 are listed but not counted, because several are summary scores or repeat a test. Each score is the one shown on the model's own page.