A lookup table for the benchmark numbers in model release announcements.
- 75 benchmarks across 10 capability domains, plus 4 aggregator sites
- Cheat sheet first (use case → benchmark), detailed tables after
- Last updated: 2026-08-07
◇ The cheat sheet
| I want a model that's good at... | Look at |
|---|---|
| broad knowledge and hard reasoning | HLE, GPQA Diamond, MMLU-Pro |
| novel-problem reasoning | ARC-AGI-2 |
| math | AIME (current year), MathArena, FrontierMath |
| writing code (functions, algorithms) | LiveCodeBench, BigCodeBench, Aider Polyglot |
| working in real codebases | SWE-bench Verified, Terminal-Bench, SWE-Lancer |
| tool calling and constrained workflows | BFCL, τ²-bench |
| web research | BrowseComp, GAIA |
| computer use | OSWorld-Verified |
| long-horizon autonomy | Vending-Bench |
| long documents | MRCR, RULER, LongBench v2, NoLiMa |
| following instructions precisely | IFBench, MultiChallenge |
| non-English languages | Global-MMLU, MMLU-ProX, FLORES |
| images and documents | MMMU-Pro, MathVista, DocVQA |
| video | Video-MME |
| creative writing | EQ-Bench Creative Writing v3, EQ-Bench Longform, TTCW |
| long fiction comprehension | NoCha, Fiction.LiveBench |
| being pleasant to talk to | LMArena, Arena-Hard |
| not hallucinating | SimpleQA, TruthfulQA |
| safety and honesty | HarmBench, AIR-Bench, MASK |
| medical use | HealthBench |
◇ The benchmarks
Status legend: saturated (frontier models at the ceiling, differences are noise), saturating (approaching the ceiling at the frontier), active (still separates frontier models).
Knowledge and reasoning
| Benchmark | Who | Year | What it tests | Size | Format | Status | Caveats |
|---|---|---|---|---|---|---|---|
| MMLU | Hendrycks et al. (UC Berkeley) | 2021 | Breadth of academic and professional knowledge recall | 15.9K Q | 4-choice MCQ, 57 subjects | Saturated | Known contamination; scores drop double digits on cleaned variants |
| MMLU-Pro | TIGER-Lab (U. Waterloo) | 2024 | Knowledge plus reasoning under harder distractors | 12.1K Q | 10-choice MCQ, 14 domains | Saturating | Inherits some MMLU contamination |
| GPQA Diamond | Rein et al. (NYU) | 2023 | Graduate-level science reasoning that web search can't shortcut | 198 Q | 4-choice MCQ, bio/phys/chem | Saturating | Small set; 1–2 pt gaps are noise |
| HLE (Humanity's Last Exam) | CAIS + Scale AI | 2025 | Expert-level knowledge at the frontier of many fields | 2,500 Q | MCQ + short answer, text + images, LLM-graded | Active | Expensive to run; judge sensitivity |
| ARC-AGI-2 | F. Chollet / ARC Prize | 2025 | Abstracting a rule from few examples, applying it to a novel case | 120 eval tasks | Grid-transformation puzzles, exact-match output grid | Active | Private test set; program search can score without "reasoning" |
| SimpleQA | OpenAI | 2024 | Factual recall, and declining to answer instead of hallucinating | 4,326 Q | Short answer, exact match | Active | Answer-type bias, some wrong ground truths; use Verified subset |
| TruthfulQA | Lin et al. (Oxford + OpenAI) | 2021 | Resistance to repeating common human misconceptions | 817 Q | MCQ + free response, judged | Saturating | Aged judge methodology; US-centric |
| HellaSwag | Zellers et al. (UW + AI2) | 2019 | Commonsense inference about what happens next | 70K Q | 4-choice sentence completion | Saturated | Small-model comparison only |
| WinoGrande | Sakaguchi et al. (AI2) | 2020 | Commonsense pronoun resolution | 44K Q | Binary fill-in-the-blank | Saturated | Small-model comparison only |
| BBH (BIG-Bench Hard) | Suzgun et al. (Google) | 2022 | Multi-step reasoning on tasks models used to fail | 6.5K Q | 23 task suites, mixed formats | Saturated | Small-model comparison only |
| DROP | Dua et al. (AI2 + UC Irvine) | 2019 | Discrete reasoning over text: counting, arithmetic, sorting | 96K Q | Free-form QA over paragraphs, F1 | Saturated | Small-model comparison only |
Math
| Benchmark | Who | Year | What it tests | Size | Format | Status | Caveats |
|---|---|---|---|---|---|---|---|
| GSM8K | OpenAI | 2021 | Multi-step arithmetic reasoning on word problems | 8.5K Q (1,319 test) | Free response, numeric exact match | Saturated | Heavily contaminated; perturbed variants tank scores |
| MATH | Hendrycks et al. (UC Berkeley) | 2021 | Competition problem solving up to AMC/AIME level | 12.5K Q (500 test) | Free response, boxed-answer exact match | Saturating | Contaminated; replaced by AIME for frontier eval |
| AIME | MAA | 2024+ | Competition math on problems too recent to have been memorized | 30 Q/year | Free response, integer answer 0–999, exact match | Saturating | Tiny sample; top models in the 90s |
| MathArena | ETH Zürich | 2025 | Generalization to brand-new competition problems, including proofs | 6–30 Q/competition | Final answers + judged proofs; sourced live from AIME, HMMT, USAMO, Putnam | Active | Small sets; proof grading partly manual |
| FrontierMath | Epoch AI | 2024 | Research-level mathematical problem solving on unpublished problems | ~350 Q | Free response, auto-checked + expert review, tiered difficulty | Active | OpenAI funding/access controversy; expert grading |
| OlympiadBench | Tsinghua + OpenBMB | 2024 | Olympiad-level math and physics, partly from diagrams | 8.5K Q | Free response + proofs, text + diagrams, EN/ZH | Active | Diagram problems mix in vision capability |
| PutnamBench | Tsoukalas et al. (UT Austin) | 2024 | Producing machine-checkable formal proofs | 1,724 problems | Formal proof generation, verifier pass/fail, Lean/Isabelle/Coq | Active | Formalization skill; not comparable to informal benchmarks |
| miniF2F | Zheng et al. (OpenAI) | 2021 | Formal proving of olympiad-level statements | 488 problems | Formal proof generation, verifier pass/fail, Lean/Isabelle | Saturated | Superseded by larger formal suites |
Coding
| Benchmark | Who | Year | What it tests | Size | Format | Status | Caveats |
|---|---|---|---|---|---|---|---|
| HumanEval | OpenAI | 2021 | Writing a standalone function from a docstring | 164 tasks | Function synthesis from docstring, unit tests, pass@k, Python | Saturated | Contaminated; thin tests (use EvalPlus variant) |
| MBPP | 2021 | Writing basic functions from short descriptions | 974 tasks | Function synthesis from description, 3 unit tests each, pass@k, Python | Saturated | Contaminated; thin tests (use EvalPlus variant) | |
| SWE-bench Verified | Jimenez et al. (Princeton) | 2024 | Resolving real GitHub issues: repo navigation, multi-file edits, tests pass | 500 tasks | Issue + repo instances, patch vs. test suite, Python | Active | Python-heavy; solutions public; harness moves scores several points |
| Multi-SWE-bench | ByteDance | 2024 | Resolving real GitHub issues across languages | 1,632 tasks | Issue + repo instances, patch vs. test suite, 7 languages | Active | Uneven difficulty across languages |
| LiveCodeBench | Jain et al. (UC Berkeley) | 2024 | Algorithmic problem solving on post-cutoff contest problems | ~600 Q, growing | Program synthesis, hidden test execution; sourced from LeetCode/AtCoder/Codeforces | Active | Algorithmic skew, not production code |
| BigCodeBench | BigCode project | 2024 | Following complex instructions through unfamiliar library APIs | 1,140 tasks | Function synthesis, unit tests, pass@1 | Active | Library-version drift |
| Aider Polyglot | Aider project (P. Gauthier) | 2024 | Editing existing code correctly inside a tool loop | 225 tasks | Code edits, unit tests, 2 attempts, 6 languages; sourced from Exercism | Active | Measures model + Aider's tooling together |
| Terminal-Bench | Stanford + Laude Institute | 2025 | Completing debugging, sysadmin and build tasks in a real shell | 89 tasks | Containerized shell, pass/fail | Active | Small task count |
| SWE-Lancer | OpenAI | 2025 | Completing paid freelance jobs end-to-end | 1,400 tasks | End-to-end jobs + manager decisions, tests + human review, dollar-weighted ($1M total); sourced from Upwork | Active | Human-judgment grading; Upwork task distribution |
| SciCode | Tian et al. (UIUC + Argonne) | 2024 | Implementing computations from scientific papers | 80 problems (338 steps) | Code synthesis from paper-derived specs, per-step unit tests, Python | Active | Domain-specific; test coverage varies |
Agents and tool use
| Benchmark | Who | Year | What it tests | Size | Format | Status | Caveats |
|---|---|---|---|---|---|---|---|
| τ-bench / τ² | Sierra | 2024 | Multi-turn tool use under policy constraints, with a simulated user | 165 tasks | Retail/airline/telecom dialogues; pass^k over reruns | Active | Three domains only |
| BFCL | UC Berkeley (Gorilla) | 2024 | Emitting correct function calls: selection, arguments, serial/parallel | ~2K calls | Single/parallel call generation, AST match against reference | Active | Mechanics only, no planning; no real execution |
| GAIA | Mialon et al. (Meta AI + Hugging Face) | 2023 | Chaining search, files and reasoning to answer assistant-style questions | 466 Q | Free-form QA with tool use, exact match | Saturating | Web answer leakage |
| BrowseComp | OpenAI | 2025 | Persistent multi-hop web research for hard-to-find facts | 1,266 Q | Short-answer QA, graded exact match, live web | Active | Live web = non-deterministic |
| WebArena (Verified) | Zhou et al. (CMU) | 2024 | Completing tasks on websites, judged by final state | 812 tasks | Self-hosted sites, execution-scored | Active | Original grading had bugs; use Verified numbers |
| OSWorld (Verified) | XLANG Lab (HKU) | 2024 | Operating a real desktop OS from screenshots | 369 tasks | Ubuntu/Windows/macOS VMs, execution-scored | Active | OS non-determinism; harness-sensitive |
| Vending-Bench | Andon Labs | 2025 | Staying coherent on one goal over hundreds of turns | 1 scenario × 365 sim days | Simulated business, scored by net worth | Active | Huge run-to-run variance (itself the finding) |
| MCP-Bench | Accenture | 2025 | Orchestrating multi-step tasks across MCP tool servers | 28 servers, 250 tools | Multi-step tool tasks, judged on completion + planning quality | Active | Young; competing suites; server-dependent scores |
Long context and instruction following
| Benchmark | Who | Year | What it tests | Size | Format | Status | Caveats |
|---|---|---|---|---|---|---|---|
| Needle-in-a-Haystack | G. Kamradt | 2023 | Verbatim retrieval of one planted sentence | parametric | 1 needle, swept position × length | Saturated | Table stakes; passing it proves little |
| RULER | NVIDIA | 2024 | Retrieval, variable tracking and aggregation as context grows | 13 tasks | Synthetic retrieval/tracking/aggregation, exact match, 4K–256K tokens | Active | Shows effective vs. advertised context; synthetic |
| MRCR | Google DeepMind (open version: OpenAI) | 2025 | Ordered retrieval of repeated items in long conversations | 2–8 needles | Ordered needle retrieval in generated dialogue, exact match, up to 1M tokens | Active | Artificial task shape |
| NoLiMa | Adobe Research | 2025 | Semantic retrieval with no keyword overlap between query and target | parametric | Needle QA, 1K–128K tokens | Active | Scores collapse past 32K for most models |
| LongBench v2 | Tsinghua | 2024 | Comprehension of realistic long material: docs, repos, dialogues | 503 Q | MCQ, contexts 8K–2M words | Active | Multiple-choice format |
| Fiction.LiveBench | Fiction.live | 2025 | Tracking plot and characters across long serialized fiction | rotating set | QA over ongoing stories | Active | Fiction only; independent of labs |
| NoCha | Karpinska et al. (UMass Amherst) | 2024 | Verifying claims about entire recently published novels | 1,001 claim pairs / 67 novels | True/false claim pairs, both must be correct | Active | Recent novels resist contamination; models near chance at release |
| IFEval | Zhou et al. (Google) | 2023 | Compliance with mechanically checkable output constraints | 541 prompts | 25 constraint types, regex-scored | Saturated | Checkable constraints only |
| IFBench | Allen Institute for AI (Ai2) | 2025 | Generalizing instruction following to unseen constraint types | 294 prompts | 58 held-out constraints, auto-verified | Active | Small set |
| MultiChallenge | Scale AI | 2025 | Retaining and honoring instructions given earlier in a conversation | 266 Q | Multi-turn dialogues, rubric + LLM judge | Active | LLM-judge scoring |
Multilingual
| Benchmark | Who | Year | What it tests | Size | Format | Status | Caveats |
|---|---|---|---|---|---|---|---|
| MMMLU | OpenAI | 2024 | How much English-language capability survives translation | 14K Q × 14 lang | 4-choice MCQ, human-translated from MMLU | Active | Inherits MMLU issues |
| Global-MMLU | Cohere For AI | 2024 | Cross-lingual knowledge, with cultural bias made visible | 14K Q × 42 lang | 4-choice MCQ, translated from MMLU, cultural-sensitivity annotations | Active | Translation quality varies by language |
| MMLU-ProX | Xuan et al. (U. Tokyo + RIKEN) | 2025 | Cross-lingual knowledge and reasoning on a harder question set | 12K Q × 29 lang | 10-choice MCQ, translated from MMLU-Pro, expert-reviewed | Active | Fewer languages than Global-MMLU |
| FLORES | Meta | 2022+ | Translation quality | 2K sent × 204 lang | Parallel corpus, BLEU/chrF | Active | MT-metric limitations |
Multimodal
| Benchmark | Who | Year | What it tests | Size | Format | Status | Caveats |
|---|---|---|---|---|---|---|---|
| MMMU | Yue et al. | 2023 | Reasoning over images that carry the content: charts, diagrams, structures | 11.5K Q | MCQ, 30 subjects | Saturating | Quality varies across domains |
| MMMU-Pro | Yue et al. | 2024 | Multimodal reasoning, hardened against text-only shortcuts | 3,460 Q | 10-choice MCQ, vision-required variant | Saturating | Frontier spread now small |
| MathVista | Lu et al. (UCLA + Microsoft) | 2023 | Mathematical reasoning over figures and plots | 6,141 Q | MCQ + free response | Active | Conflates vision and math failures |
| ChartQA | Masry et al. (York U.) | 2022 | Extracting and reasoning over data in charts | 32.7K Q | Open-ended QA on real-world charts, relaxed exact match | Saturated | Synthetic questions much easier than human-written |
| DocVQA | Mathew et al. (IIIT Hyderabad + CVC) | 2020 | Reading scanned documents: OCR plus layout understanding | 50K Q / 12.7K docs | Extractive QA on document images, string-similarity scored (ANLS) | Saturating | More OCR than reasoning |
| Video-MME | Nanjing University et al. | 2024 | Temporal understanding of video | 2,700 Q / 900 videos | 4-choice MCQ on videos 11s–1h | Active | Correlates with context window length |
Human preference and arenas
| Benchmark | Who | Year | What it tests | Size | Format | Status | Caveats |
|---|---|---|---|---|---|---|---|
| LMArena (Chatbot Arena) | LMSYS / UC Berkeley | 2023 | Which model's answers real users prefer, blind | 6M+ votes | Live pairwise votes → Elo | Active | Preference, not capability; rewards verbosity/formatting |
| Arena-Hard | LMSYS | 2024 | Answer quality on the hardest arena prompts | 500 prompts | Open-ended prompts, pairwise LLM judge vs. GPT-4 baseline | Active | Judge bias; over-represents coding/math |
| AlpacaEval 2.0 | Dubois et al. (Stanford) | 2024 | Instruction-following quality | 805 prompts | Length-controlled win rate vs. GPT-4 | Active | Circular for GPT-4-family models; cheap to run |
| MT-Bench | LMSYS | 2023 | Multi-turn answer quality | 80 Q | Two-turn questions, GPT-4-judged 1–10 | Saturated | Tiny; survives as a calibration set |
Creative writing
| Benchmark | Who | Year | What it tests | Size | Format | Status | Caveats |
|---|---|---|---|---|---|---|---|
| EQ-Bench Creative Writing v3 | S. Paech (EQ-Bench) | 2025 | Short-form creative prose quality: voice, humor, unusual perspectives | 32 prompts × 3 runs | Free generation, LLM-judged rubric + pairwise Elo | Active | Single LLM judge; judge taste leaks into rankings |
| EQ-Bench Longform Writing | S. Paech (EQ-Bench) | 2025 | Sustaining a planned novella: coherence, character consistency, restraint | 8 chapters × ~1K words | Chapter-by-chapter generation, 14-criterion LLM rubric | Active | Single LLM judge; penalizes specific formatting patterns |
| WritingBench | X-PLUG (Alibaba) | 2025 | Writing across domains, literature included | 1,239 prompts | Free generation, per-query criteria, fine-tuned critic model | Active | Critic ~83% human agreement; only partly creative |
| LitBench | Fein et al. (Stanford) | 2025 | Whether judges and reward models match human story preferences | 2,480 test pairs | Human preference pairs from r/WritingPrompts | Active | Evaluates evaluators, not generators; Reddit taste as ground truth |
| TTCW | Chakrabarty et al. (Columbia + Salesforce) | 2024 | Story creativity along Torrance dimensions: fluency, flexibility, originality, elaboration | 14 binary tests | Expert human evaluation, consensual assessment | Active | Needs human experts; LLM judges don't correlate with them |
| BuzzBench | S. Paech (EQ-Bench) | 2024 | Humor understanding: explaining joke mechanics, predicting funniness | fixed prompt set | Free-form analysis, LLM-judged rubric, 5–10 runs | Active | Niche source material (UK panel show) |
| WebNovelBench | Lin et al. | 2025 | Long-form storytelling from a synopsis | 4,000 synopses | Free generation, LLM-judged on 8 dimensions, percentile rank | Active | Chinese web-novel domain; LLM judge |
| Hemingway-bench | Surge AI | 2025 | Writing quality as professional writers judge it: taste, originality, coherence | undisclosed | Free generation, expert human judging | Active | Prompt set and size not public |
| LMArena creative-writing category | LMSYS / UC Berkeley | 2023+ | Which model's creative writing users prefer, blind | subset of arena votes | Live pairwise votes → Elo | Active | Same verbosity/formatting biases as main arena |
Safety
| Benchmark | Who | Year | What it tests | Size | Format | Status | Caveats |
|---|---|---|---|---|---|---|---|
| HarmBench | Mazeika et al. (CAIS) | 2024 | Whether direct requests and automated jailbreaks elicit harmful output | 510 behaviors | Direct + adversarial variants, attack success rate | Active | Jailbreak arms race dates the attack set |
| JailbreakBench | Chao et al. (UPenn) | 2024 | Attack success and defense effectiveness | 100 behaviors | Jailbreak attempts vs. behavior list, attack success rate; rolling artifacts | Active | Moving target, hard to compare over time |
| AIR-Bench 2024 | Stanford CRFM | 2024 | Refusal behavior across regulation-derived risk categories | 5,694 prompts | Refusal probes, graded severity, 314 risk categories | Active | Jurisdiction-specific (US/EU skew) |
| HealthBench | OpenAI + 262 physicians | 2025 | Clinical quality and safety of health conversations | 5,000 convs | Graded against 48K physician rubric criteria | Active | Medical only; rubric quality = physician panel quality |
| MASK | CAIS + Scale AI | 2025 | Whether the model states things it "believes" false under pressure | 1,000+ scenarios | Pressure prompts checked against elicited beliefs | Active | Newer, smaller adoption |
Aggregators
| Site | Who | What it does | Caveats |
|---|---|---|---|
| Artificial Analysis | artificialanalysis.ai | Weighted intelligence index (~10 benchmarks) + price/speed data | Weighting is editorial |
| Epoch AI Benchmarking Hub | Epoch AI | Tracks 100+ benchmarks, capability trends over time | Aggregation methodology opaque |
| HELM | Stanford CRFM | Multi-dimensional profiles: accuracy, robustness, fairness, efficiency | No single score; slow to add new models |
| LiveBench | Abacus.AI + NYU et al. | Monthly-refreshed questions, objective grading only | Month-to-month scores not directly comparable |
◇ Reading notes
- Public benchmarks leak into training data, so a high score can mean memorization (contamination)
- Frontier models eventually cluster at a benchmark's ceiling, after which score differences are noise (saturation)
- Agentic scores depend on the evaluation harness as much as the model; that's what the "Verified" editions fix
- A lab scoring well on its own benchmark, graded by a judge it picked, is weaker evidence than an externally run leaderboard