Major AI Benchmarks Explained: What Those AI Scores Actually Measure
Ten major AI/LLM capability benchmarks (MMLU, GPQA, SWE-bench, HumanEval, AIME, etc.) compared on what they measure, format/size, scoring and whether they are saturated — to make sense of the scores models claim.
| Benchmark | What it measures | Format / size | Scoring | Status |
|---|---|---|---|---|
| MMLU | Knowledge across 57 subjects | Multiple-choice, ~14,000 (test) | Accuracy % | Near saturation at frontier |
| MMLU-Pro | Harder, more reasoning-heavy MMLU | 10-choice, 12,032 items | Accuracy % | Still discriminative |
| GPQA (Diamond) | Grad-level bio/phys/chem (Google-proof) | 4-choice, Diamond 198 | Accuracy % (25% random) | Among hardest knowledge tests |
| SWE-bench (Verified) | Fix real GitHub issues | Verified 500 instances | Resolve rate % | Not saturated; main SWE bench |
| HumanEval | Python function generation | 164 items, run unit tests | pass@1 | Saturated; contamination concerns |
| AIME | Hard competition math | 30 items/yr (answers 0–999) | Exact match % | Common for reasoning; not saturated |
| MATH | Competition-level math solving | 12,500 items, levels 1–5 | Accuracy % | Near saturation |
| GSM8K | Grade-school multi-step word problems | ~1,000 (test) | Exact match % | Near saturation |
| HellaSwag | Commonsense sentence completion | 4-choice, ~10,000 (val) | Accuracy % | Saturated; losing signal |
| MMMU | College-level multimodal (image+text) | 10,500 (test), 30 subjects | Accuracy % | Key multimodal benchmark |
Method & sources
Each benchmark's definition, format and size follow its original paper (arXiv) or official repo/site (sources cited per row). ⚠ The "status/saturation" column is qualitative — specific "XX% reached" saturation figures mostly come from secondary aggregators, not the original papers or live official data, so this table omits those percentages and keeps only verifiable qualitative status (e.g. "near saturation", "not yet saturated"). Sizes note the split (e.g. MMLU test ~14,000 items, SWE-bench Verified 500 instances). AIME is borrowed from a math competition with no single official AI-benchmark paper, as noted. Benchmark definitions are relatively stable, though the community releases harder successor versions; refer to official sources.
Source: https://arxiv.org/abs/2009.03300
Retrieved: 2026-07-26