WikiBeck

Comparing LLMs by coding benchmark

4 min read

Comparing LLMs by coding benchmark

Public LLM leaderboards rank models by different benchmarks that measure different things. Reading a leaderboard usefully means knowing what each column actually tests, not just which model is “#1” today — rankings from this space change weekly and any specific numbers go stale almost immediately.

How it works

Common categories of coding/agentic benchmark seen on leaderboards:

  • Aggregate coding scores (e.g. a weighted blend of an agentic-fix benchmark and a competitive-programming benchmark): meant to approximate “how good is this model at realistic coding tasks end to end,” but the exact weighting varies by provider and changes over time — treat the number as an ordinal ranking signal, not an absolute score comparable across sites.
  • SWE-bench-style benchmarks: measure whether a model can resolve real, verified GitHub issues end to end (read an issue, patch a real codebase, pass the existing test suite). Closer to actual agentic coding work than pure code-generation benchmarks, but scores vary a lot by scaffolding (the harness/tool access given to the model), so cross-provider comparisons on this number alone can be misleading unless the harness is held constant.
  • Arena / human-preference ratings: crowd-sourced pairwise comparisons voted on by users, aggregated into an Elo-style rating. Captures subjective quality and instruction-following, not narrow task-completion accuracy — a model can rank high here and mediocre on SWE-bench, or vice versa.
  • Price per 1M tokens (input/output): the input price matters most for large-context/RAG-heavy workloads; output price dominates for long, reasoning-heavy responses. Compare price and benchmark together — a much cheaper model that’s only slightly worse is often the better default, and a “top of leaderboard” model is frequently 5-10x the cost of one a few points behind.

When to use it / trade-offs

Use a leaderboard to shortlist 2-4 candidates for a specific workload, then verify with your own task samples — none of these benchmarks are perfect proxies for a specific use case (e.g. a model strong on SWE-bench-style issue resolution is not automatically the best choice for a chat assistant, and vice versa). Re-check pricing and rank at the point you actually need to choose; don’t rely on a snapshot more than a few weeks old.

Pitfalls

  • Self-reported vs. independently verified scores. Some numbers on leaderboards are reported by the model’s own provider and later revised (sometimes lowered) once an independent site reproduces the benchmark. Prefer numbers from a site that verifies runs itself, and be skeptical of numbers that only ever appear on the provider’s own materials.
  • Whole-board rescales, not just per-model updates. A leaderboard operator can revise its scoring methodology and re-verify the entire board at once, shifting most models’ scores by several points in the same refresh (observed: a mid-2026 rescale moved many scores down 1-5 points board-wide, with a few outliers dropping 15+ points). A big score drop after such an event does not necessarily mean the model regressed — check whether the refresh note mentions a methodology/scoring change before reading it as a capability change.
  • Scaffolding-dependent benchmarks. SWE-bench-style and other agentic benchmarks are run with a specific tool/harness setup; the same underlying model can score very differently under a better or worse harness. A benchmark table rarely tells you which harness was used.
  • Context length ≠ usable context. A large advertised context window doesn’t mean quality holds up near that limit — some models degrade well before their nominal maximum. No leaderboard column captures this; test it directly for long-context use cases.
  • Weekly churn. Rankings and prices from these sources move frequently (new model releases, price cuts, benchmark re-verification). Don’t hard-code a “best model” decision into documentation — link to the live leaderboard instead of copying a table of current numbers.
  • Current LLM coding leaderboard — automatically refreshed BenchLM coding scores with OpenRouter-listed API prices; treat it as a dated snapshot, not a universal model ranking.
  • KV-cache cost — for self-hosted models, the benchmark score is only half the picture; whether it fits your hardware at a usable context length is the other half.

References

Check the live listings rather than a point-in-time copy:

Source captures (private)

Graph View

Backlinks