Gemma (7B) is the stronger all-rounder.
Scores updated · 26 tests both models report · How we compare
Where each one wins
Tests won in each of the one area where both have results. Each piece is one test, so a longer bar means more evidence; grey means the two scored within a point of each other.
Reasoning rests on a single test.
The biggest differences
The tests each model wins by the widest margin, up to three each. Scores are out of 100.
Where Gemma (7B) pulls ahead
- Graduate-level biology, physics and chemistry questionsGPQA Diamond+1points ahead
Where Mistral 7B pulls ahead
No clear win on a test scored out of 100.
Every test, side by side
All 26 tests both models report. The winning score is in its model's colour; marks a score checked independently.
Other results25 tests, not counted
Tests outside the eight areas. They are not counted above: several are summary scores built from other tests, or the same test under another name.
- MATHGemma (7B) by 36.95013.1+36.9
- Multi-MathematicsGemma (7B) by 12.839.126.3+12.8
- Social IQAMistral 7B by 9.165.574.6+9.1
- HumanEvalGemma (7B) by 7.937.229.3+7.9
- MMLUMistral 7B by 7.157.164.2+7.1
- AGI EVALGemma (7B) by 742.135.1+7
- BoolQMistral 7B by 6.26672.2+6.2
- C-EvalMistral 7B by 3.843.647.4+3.8
- GSM8KGemma (7B) by 3.755.952.2+3.7
- EvalPlusGemma (7B) by 3.239.636.4+3.2
- Popular aggregated benchmarkGemma (7B) by 2.959.456.5+2.9
- TriviaQAMistral 7B by 2.972.375.2+2.9
- MMLU-ProGemma (7B) by 2.833.730.9+2.8
- TruthfulQAGemma (7B) by 2.644.842.2+2.6
- Theorem QAGemma (7B) by 2.321.519.2+2.3
- ANLIGemma (7B) by 1.648.747.1+1.6
- ARC-ChallengeGemma (7B) by 1.161.160+1.1
- BBHMistral 7B by 155.156.1+1
- HellaSwagMistral 7B by 182.283.2+1
- OpenBookQAGemma (7B) by 180.879.8+1
- WinoGrandetie7978.4tie
- MBPPtie50.651.1tie
- MedQAtie49.650tie
- PIQAtie78.177.7tie
- MultiPL-Etie29.729.4tie
Questions people ask
Which is better, Gemma (7B) or Mistral 7B?
Gemma (7B) wins the one area where both have results: reasoning. Mistral 7B wins none.
How do you compare the two?
We use the 26 benchmark tests both models have published scores on. The verdict counts the 1 tests in the eight capability areas, and a gap under one point (ten on rating-style scales) counts as a tie. The other 25 are listed but not counted, because several are summary scores or repeat a test. Each score is the one shown on the model's own page.