Benchmarks

SWE-bench (+Verified/Pro)

Tests whether an AI agent can resolve real GitHub issues by writing a patch that passes the repository's hidden test suite.

Originally Princeton & Stanford; the Verified subset was curated with OpenAI, the Pro variant is run by Scale AI.

⚠️ Independent audits found solutions that were effectively given away in the issue thread, and tests too weak to catch a broken fix — and SWE-bench Pro alone has several conflicting "official" scores depending on which scaffold and data split is used.

Terminal-Bench

Tests whether an agent can complete real multi-step terminal tasks, where every command's output changes what it should do next.

A Stanford x Laude (Anthropic-adjacent) collaboration, part of the open "Harbor" framework.

Aider Polyglot

Tests whether a model follows instructions and edits code correctly, unassisted, across coding exercises in several languages.

Run by Paul Gauthier, Aider's creator — not a lab or consortium.

⚠️ This leaderboard isn't re-run for every new model — it can show outdated models on top for months, simply because newer ones haven't been tested yet.

LiveCodeBench

Tests code generation, self-repair, predicting test output, and code execution, using competitive-programming problems pulled in continuously so old problems can't leak into training data.

Researchers at UC Berkeley, MIT, and Cornell (lead: Naman Jain).

LMArena Code Arena / WebDev Arena

Measures human preference, not correctness: real people blind-vote on which of two models' output (code or web-app) they like better.

Arena.ai (the LMArena / LMSYS project).

⚠️ A large-scale analysis of vendor submissions found that selectively submitting only the best-performing variant can meaningfully inflate a model's score.

HumanEval

Tests functional correctness on hand-written Python programming problems by actually running the generated code against unit tests.

Created by OpenAI researchers (Chen, Tworek, Jun and others).

⚠️ Saturated: nearly all frontier models cluster tightly together at the top, so the benchmark barely tells them apart anymore. It's used only as a basic sanity check now, not for ranking frontier models.

Artificial Analysis Coding Agent Index

A composite score for end-to-end coding-agent performance — combining several underlying benchmarks plus cost, token, and time efficiency into one index.

Run by Artificial Analysis, an independent benchmarking outfit.

⚠️ Scores the model-plus-harness combination together, so the same underlying model can rank very differently depending on which agent scaffold runs it.

ARC-AGI-2

Tests abstract, fluid reasoning through visual grid-transformation puzzles, plus how efficiently (at what cost) a model solves them. Not a coding benchmark per se.

Run by ARC Prize Inc., a nonprofit (keeps test data private or semi-private to resist contamination).

MMLU / MMLU-Pro

Tests general knowledge and reasoning with multiple-choice questions spanning many academic subjects. Not coding-specific.

Originally created by Hendrycks and colleagues (UC Berkeley-affiliated researchers).

⚠️ Saturated: frontier models cluster too closely together to differentiate. MMLU-Pro was built to fix that, but is reported to be starting to saturate too.

METR "Time Horizon"

Tracks how long a task (in human-expert work-time) an agent can complete on its own at a fixed reliability bar — read as a trend line over time, not a single score.

Run by METR, an independent AI-evaluation nonprofit.

⚠️ METR itself flags its measurements for very long tasks as unreliable with the current task suite — read it as a trend line, not a precise cutoff.