Leaderboard Design: Ranking Without Gaming

Goodhart's Law Comes for AI Leaderboards

When a ranking starts carrying real stakes — funding, headlines, business deals — it stops being a measurement and starts being a target. That's Goodhart's Law in action: when a measure becomes a target, it ceases to be a good measure, and the AI industry's flagship leaderboard just provided the most vivid case study yet.

In spring 2025, a 68-page analysis from researchers at Cohere, Stanford, MIT, and the Allen Institute for AI — dubbed the "Leaderboard Illusion" paper — documented how a handful of large providers allegedly gamed LMArena, the Chatbot Arena rankings that much of the industry treats as ground truth. The findings were striking: Meta reportedly tested 27 private model variants before the Llama 4 release, disclosing only the best performer, and the authors estimated that increased access to Arena data could inflate a model's ranking performance by up to 112%. LMArena's co-founder called the study "full of inaccuracies," and Google disputed the variant counts — though notably, the rebuttals reportedly did not contest the underlying data analysis.

The uncomfortable takeaway is that gaming was structural, not incidental. A model's ranking shapes its market value, drives investor decisions, and influences regulation, so the incentive to bend the measurement is enormous. And distortion doesn't require bad actors. Stanford researchers analyzing 56 widely used AI benchmarks found that tests claiming to measure the same construct frequently disagree — their BBQ bias benchmark example showed a model that spots trick questions scoring as unbiased while an honestly unbiased model that misses the trick scores as biased. The measuring stick itself was broken.

Even the math is fragile. University of Michigan researchers demonstrated that the ranking algorithm alone changes outcomes on identical data: Bradley-Terry is accurate only with even comparison counts per model, while Elo and Markov Chains are highly configuration-dependent. Chess-style assumptions simply don't transfer to AI models, which don't change between games and can play millions simultaneously.

For anyone building or trusting a ranking system, the remedies converge across sources:

• Publish all evaluation runs permanently, with no retraction or selective disclosure
• Apply uniform testing conditions to every participant — no private variant testing
• Use multiple diverse metrics rather than a single composite score
• Test on fresh, unseen material to defeat contamination and teaching-to-the-test
• Calibrate scores against real-world outcomes, which Stanford calls the missing piece

These findings are exactly why SkillzBench, AIQ Labs' independent testing lab for agent skills, is designing its benchmarks around them from day one: every skill tested in a sandbox with the same model, with and without the skill, and the resulting Lift published in percentage points — always shown separately from trust ratings, with money never touching scores. In a skill ecosystem where published research found roughly 26% of skills carry at least one vulnerability, a ranking you can actually trust isn't a nice-to-have. It's the whole point.

Skills, proven. Join the SkillzBench waitlist to see the Lift for yourself before you install another skill.

Why Rankings Break Even Without Bad Actors

You don't need cheaters to break a leaderboard. The machinery itself can fail quietly — same data, different ranking algorithm, different winner.

University of Michigan researchers studying AI leaderboards found that the ranking method itself changes results on identical data. Glicko was the most consistent, especially when models had uneven numbers of comparisons. Bradley-Terry was accurate only with even comparisons per model, while Elo and Markov chains proved highly configuration-dependent. Chess-style assumptions don't transfer, either — AI models don't change between releases and can play many games simultaneously.

Then there's benchmark contamination. Datasets get swept into training data, so models "ace tests they've essentially seen before," as IBM researchers put it. The model looks brilliant while demonstrating only memory. That's why the industry is shifting toward dynamic evaluation — IBM's LiveXiv refreshes monthly with fresh arXiv material, and Amazon's SWE-PolyBench found performance dropped sharply outside Python.

Construct confusion is quieter still. Across 56 widely used benchmarks, Stanford researchers found that tests claiming to measure the same construct frequently disagree. The BBQ bias benchmark is the defining example: a biased model that spots trick questions scores as unbiased, while an unbiased model that misses the trick scores as biased. The test measures question-spotting, not fairness.

These are structural problems, not intention problems. No amount of good faith fixes an algorithm mismatch or a memorized test set. Design-level remedies are required:

• Publish all evaluation runs permanently, with no retraction or selective disclosure
• Test every entry under identical conditions — same model, same tasks, same protocol
• Rotate benchmark material so static tests can't be memorized
• Report multiple metrics separately rather than collapsing everything into one composite score

This is the design philosophy behind SkillzBench, AIQ Labs' independent testing lab for agent skills. Every skill is benchmarked in a sandbox with the same model, with and without the skill, and the resulting Lift — in percentage points — is published separately from the 0-100 trust score. Money never touches scores, and every version is pinned to a commit SHA so results can't be quietly re-run until favorable. SkillzBench is building this now; the waitlist is open for teams that want benchmark results they can actually trust.

As Stanford's Sanmi Koyejo warns, "Whoever holds the measuring stick steers the ship… if you get the evaluation wrong, the risks are quite great and rising." Getting the measuring stick right is a design decision — and it has to be made before the leaderboard goes live.

The Anti-Gaming Playbook: Five Design Patterns That Work

The LMArena controversy showed what happens when a leaderboard becomes a target: Meta reportedly tested 27 private model variants before the Llama-4 release, with undisclosed score retraction, and increased data access could boost Arena performance by up to 112%. The remedies converge across sources into a playbook any honest ranking system can adopt.

Publish everything, permanently. Cohere's Sara Hooker argues that all evaluation results — including private variants and unflattering runs — should be permanently published with no retraction, because "it is critical for scientific integrity that we trust our measure of progress." Selective disclosure is the root distortion; a leaderboard that lets participants hide failed runs invites re-running until something favorable sticks.

Test everyone identically. The documented gaming came from asymmetric access, not just bad data. Uniform conditions for every participant — same model, same sandbox, same protocol — close that door. This is the principle behind SkillzBench's benchmarking approach: testing every agent skill with the same model, with and without the skill, and publishing the Lift in percentage points, applied the same way to every skill in the catalog.

Split the score. Single composite numbers are Goodhart bait. Stanford researchers analyzing 56 widely used benchmarks found that tests claiming to measure the same construct frequently disagree — and their BBQ bias example shows how one number can invert the truth: a biased model that spots trick questions scores as unbiased. Pairing metrics helps. SkillzBench separates its 0–100 trust score (built from security, quality, license, and maintenance signals) from Lift, so improvement is never laundered through a single number.

Fight contamination with fresh tests. Static benchmarks get swept into training data, so models "ace tests they've essentially seen before," as IBM's evaluation researchers documented — their LiveXiv refreshes monthly with fresh arXiv material, and their MTRAG suite includes intentionally unanswerable questions to catch bluffing. Rotating tasks and re-benchmarking on upstream changes keeps scores measuring current capability, not memorization.

Calibrate against reality. Stanford identifies calibration of scores against real-world outcomes as "the missing piece" in benchmark design. Andrej Karpathy points to OpenRouter's usage-based rankings as "very difficult to game" precisely because they track what people actually choose. As IBM's Gabe Goodhart puts it: "The real test isn't if a model scores well on a test — it's whether it helps you get something done."

Together, these five patterns — permanent publication, uniform conditions, diverse metrics, fresh tasks, and real-world calibration — turn a leaderboard from a gaming target back into a trustworthy measure of progress.

Designing Leaderboards That Motivate Instead of Demoralize

Most leaderboards quietly lose their audience. Research on gamification in education found that while leaderboards influence learner motivation more than badges or prizes, they can also push lower-ranked participants to disengage, with negative experiences triggering "downward counterfactual thinking" that erodes confidence (JMIR Serious Games).

The problem is structural, not cosmetic. Park and Kim's analysis of 754 gamification cases found that macro-only leaderboards — single, overall rankings — induce only "fragmented competition," leaving participants outside the top ranks feeling deprived of any meaningful path to recognition (their study). A leaderboard that shows one winner-takes-all list isn't motivating; it's a weekly reminder to most participants that they've lost.

The remedy the research points to is pairing macro and micro leaderboards. Instead of one overall rank, publish per-category standings where different participants can lead different dimensions, and incorporate all achievement measures into the micro view so progress is visible even when absolute rank hasn't changed. Framing results around improvement — how much better someone performed against their own baseline — rather than absolute position keeps mid-pack participants engaged instead of demoralized.

• Publish per-category leaderboards alongside any overall ranking, so participants can compete where they're strongest.
• Incorporate every achievement measure into the micro leaderboard, not just the headline score.
• Frame results as improvement against baseline rather than winner-takes-all rank.
• Avoid single composite scores, which invite both gaming and measurement confusion.

These principles matter well beyond classrooms. In AI evaluation, Stanford researchers found that 56 widely used benchmarks claiming to measure the same construct frequently disagree, and a single biased score can misrepresent a model entirely (Stanford HAI). Multiple diverse metrics beat one number — which is why SkillzBench plans to publish Lift in percentage points separately from its 0–100 trust score, so a skill's measured improvement is never blended into a rating that money or community sentiment could touch.

The same motivation logic applies to how results are framed. SkillzBench's benchmark protocol — testing every skill in a sandbox with the same model, with and without the skill — produces a Lift figure that is inherently an improvement measure: how much the skill helped versus baseline. Presenting those results as "Lift vs. baseline" per category, rather than an absolute rank order, gives every skill author a meaningful read on their work, whether they sit at the top of a category or the middle.

A leaderboard designed this way engages. A macro-only, composite-score leaderboard — the research is clear — drives people away.

How SkillzBench Ranks Agent Skills Without Gaming

Designing a leaderboard that can't be gamed is harder than it looks — the research above shows even the AI industry's flagship rankings bent under pressure. SkillzBench is applying those lessons directly, and the result is a set of design commitments built to survive Goodhart's Law rather than discover it the hard way.

The first commitment is separating Lift from trust. Every skill gets benchmarked in a sandbox with the same model, run twice — once with the skill, once without — and the difference is published as Lift in percentage points. That number stays visibly separate from the 0–100 trust score, which draws from security, quality, license, maintenance and community signals. This mirrors the research finding that single composite scores invite both gaming and construct confusion — Stanford's analysis of 56 widely used benchmarks found that tests claiming to measure the same thing frequently disagree, and blending metrics makes the confusion worse (Stanford HAI).

The second commitment is uniform testing. The LMArena controversy showed how asymmetric access distorts rankings — one provider reportedly tested 27 private model variants before a release, and researchers estimated data-access advantages worth up to 112% in Arena performance (Collinear AI's analysis). SkillzBench's answer is identical conditions for every skill, no private re-runs until a favorable number appears.

The full anti-gaming playbook SkillzBench is building:

• Money never touches scores. Paid tiers buy convenience, never ranking position.
• Every version pinned to a commit SHA, with runs permanently published — no retraction, no selective disclosure.
• Task rotation and re-benchmarking on upstream changes, because static tests get memorized.
• Per-category Lift leaderboards alongside any overall view, framing results as improvement over baseline rather than winner-takes-all rank.

That last point comes straight from the gamification literature: macro-only leaderboards induce what researchers call "fragmented competition" and demotivate anyone below the top (JMIR Serious Games). A skill that adds five points of Lift in code review is genuinely useful even if another skill adds twelve in data analysis. IBM's Gabe Goodhart put the philosophy plainly: the real test "isn't if a model scores well on a test — it's whether it helps you get something done" (IBM Think). Lift measures exactly that.

These are commitments in active development, not finished claims — SkillzBench is pre-launch, and the benchmarking pipeline is what it is building toward. If you want to see whether a skill actually helps before you install it, join the waitlist at skillzbench.com. Skills, proven — that's the whole point.