Why Benchmark Scores Lie: The Reproducibility and Contamination Crisis

That 92% on the leaderboard might be the least true number in your AI stack. The benchmark scores you see published for LLMs are quietly shaped by two forces most readers never hear about: fragile implementation details and training-data contamination.

Start with reproducibility. MMLU can be implemented at least three ways — comparing probability of option tokens A–D, including all possible tokens, or comparing full-text sequence probability — and research from Together AI found different institutions using these approaches got different results and even different model rankings. As the authors put it, conclusions about model capabilities might depend on implementation details rather than actual performance differences.

The details that swing scores sound almost trivial: how many few-shot examples you include, whether log-likelihoods are length-normalized, and how the prompt is formatted. Practitioner guidance on lm-evaluation-harness notes these choices can shift reported scores by several points, and prompt formatting is the single biggest culprit — omitting the chat template for a chat model produces "garbage scores because the prompt is malformed for that model." A 0-shot MMLU number and a 5-shot MMLU number are, functionally, different benchmarks.

Contamination cuts even deeper. Benchmark data can leak into training corpora, inflating scores on problems the model has effectively memorized. The evidence is stark:

• On GSM1K — a fresh set designed to mirror GSM8K — many models scored significantly lower than on GSM8K itself, suggesting high GSM8K scores partly reflect memorization rather than reasoning.
• When state-of-the-art reasoning models were evaluated on USAMO 2025 problems hours after release, before contamination was possible, even the best models scored below 5%.
• A University College Dublin survey finds the majority of academic studies agree contamination undermines the reliability and validity of LLM evaluations — and AI-generated content is making it harder to detect.

So if the raw numbers can't be trusted at face value, what should you look for in a benchmark? A trustworthy benchmark has five properties: difficulty, diversity, usefulness, reproducibility, and resistance to contamination. Difficulty means models can't ace it through pattern-matching alone; diversity means it samples enough of the capability space that one lucky subscore can't carry the result; usefulness means it measures something you actually care about.

Reproducibility means every reported number comes with its full context — model, task version, few-shot count, template settings — so anyone can re-run it. Contamination resistance means fresh or held-out data that the model can't have seen in training. These are the same principles SkillzBench is designing its skill benchmarking around: every version pinned to a commit SHA, tested in a sandbox with the same model, with and without the skill, so the Lift score reflects the skill — not the noise. When you evaluate anything in the AI stack, demand these five properties before you trust a number.

From Engine Tests to Road Tests: Benchmarking Models vs. Agents vs. Skills

An LLM that aces every static benchmark can still fail spectacularly when handed tools and a multi-step task. That gap is why evaluation experts draw a sharp line between testing a model and testing an agent. As a peer-reviewed KDD '25 survey puts it, LLM evaluation is like examining an engine, while agent evaluation assesses the whole car under real driving conditions.

Static harnesses like lm-eval measure the engine: fixed, gold-labeled datasets such as MMLU or GSM8K, scored with deterministic metrics. They are excellent for comparing base models — but they say nothing about how an application behaves. The same tutorial is explicit that harnesses are not designed for evaluating RAG pipelines, agents, or chatbots, which need application-level evaluation instead.

Agent evaluation is harder because agents are "inherently probabilistic and behave dynamically," and mistakes propagate and compound across turns. Anthropic's engineering team notes that frontier models can even fail a static eval while producing a better real-world outcome — one model found a policy loophole in a flight-booking task that failed the test as written but served the user better. Agent behavior is therefore measured across four dimensions: task completion, output quality, latency, and cost.

Grading any of this requires choosing among three grader types:

• Code-based graders — fast, cheap, objective, and reproducible, but brittle to nuance.
• Model-based graders — flexible and nuanced, but non-deterministic and prone to bias.
• Human graders — the gold standard for creativity and reasoning quality, but slow and expensive.

The emerging best practice is a hybrid. LLM-as-judge scales beautifully, but judges systematically prefer longer responses and favor outputs from their own model family, so bias correction is essential. The practical validation threshold: automate an AI judge only after it reaches 85–90% agreement with human reviewers; below that, tune it or narrow its scope.

These principles translate directly to skill evaluation. A SKILL.md instruction package isn't a model — it's a modification to an agent's behavior, so the only honest test is a controlled A/B: run the same model in a sandbox with and without the skill, and measure the difference. That's the design SkillzBench is building — every skill tested under identical conditions, with Lift published in percentage points across task success, quality, latency, and token cost, kept separate from any rating. It's the road test, not the engine dyno — and it's the only way to know whether a skill actually helps.

The Five-Step Benchmarking Playbook

Most benchmark numbers you see online aren't wrong — they're unreproducible. Tiny implementation choices can swing reported scores by several points and even change model rankings, meaning conclusions often reflect setup details rather than real capability differences, as Together AI's evaluation research shows. Here's the five-step playbook that keeps your benchmarks honest.

Step 1: Pin everything and publish a reproducibility manifest. Record the harness version, task name and version, few-shot count, chat-template usage, backend, and dtype with every number. The lm-evaluation-harness tutorial is blunt: "A 0-shot MMLU number and a 5-shot MMLU number are different benchmarks." Prompt formatting is the #1 cause of score discrepancies — omitting the chat template for chat models produces garbage scores from a malformed prompt. SkillzBench applies the same discipline to skills, pinning every version to a commit SHA before it's ever tested.

Step 2: Use paired, controlled comparisons. Test the same model with and without the variable you're isolating — nothing else changes. This A/B-style design is what separates "the skill helped" from "the model was different." It's the design SkillzBench is building its Benchmark pillar around: same model, same sandbox, with and without the skill, with the Lift published in percentage points.

Step 3: Guard against contamination. When state-of-the-art reasoning models were evaluated on USAMO 2025 hours after release, even the best scored below 5% — while familiar benchmarks show 90%+ accuracy on saturated sets like MATH. Fresh or rotating test sets are your defense; lm-eval even ships a decontamination tool to detect test-set leakage.

Step 4: Report multiple dimensions, never a single score. The KDD '25 agent-evaluation survey defines four black-box dimensions worth tracking:

• Task completion — success rate, pass@k
• Output quality — rubric-scored, not just correct
• Latency — time to first token, end-to-end
• Cost — token counts per task

A model that wins on accuracy but triples your token spend isn't a win.

Step 5: Separate capability benchmarks from regression monitoring. Anthropic's engineering guidance is clear: capability evals should start at low pass rates, while regression suites should pass near 100% and run continuously as things change upstream. Teams with evals upgrade to new models in days rather than weeks — the regression suite is what catches the breakage when a skill's upstream SKILL.md changes or a new model version shifts behavior.

That's the framework: pin, pair, rotate, multi-dimensional, and split capability from regression. Skills, proven — that's the standard SkillzBench is building toward, one pinned benchmark at a time.

Keeping Benchmarks Alive: Refresh, Re-run, and Trust the Process

Benchmarks don't stay valid on their own. MMLU took roughly four years for models to saturate, while GPQA reached similar levels in just one year — and SWE-bench Verified performance jumped from 40% to over 80% in a single year, according to Anthropic's engineering team. Any benchmarking framework that assumes its numbers stay current is already wrong.

That's why evaluation needs a lifecycle, not a one-time score. Anthropic distinguishes capability evals, which start at low pass rates and measure whether something works at all, from regression evals, which should pass near 100% and run continuously. A skill benchmark is the same: the initial run measures Lift, and every run after that guards against drift.

For agent skills, the drift trigger is concrete. When a skill author updates a SKILL.md upstream, the instruction package your agent loads has silently changed. A practical framework treats that as a regression event:

• Pin every skill version to a commit SHA so you always know exactly what was tested.
• Re-run the same benchmark suite against the new version, treating the original result as the fixed baseline.
• Publish the Lift delta in percentage points, not just a fresh score, so readers see what changed.
• Notify users whose agents load the affected skill.

Automation alone isn't enough to keep this honest. LLM judges exhibit measurable biases — favoring longer responses and outputs from their own model family — and should only be trusted for automated use after reaching 85–90% agreement with human reviewers. Human oversight remains what catches the failures automation misses, from subtle reasoning errors to novel failure modes a judge has never seen.

This lifecycle thinking is what SkillzBench is building. Every skill will be benchmarked in a sandbox with the same model, with and without the skill, and the resulting Lift published in percentage points — with trust scores kept strictly separate from Lift, and money never touching scores. Upstream changes trigger re-scans, re-benchmarks, and notifications, so a skill's number reflects the version you're actually running.

If you want to know which skills actually help your agents — and which ones quietly stopped working after an upstream edit — join the SkillzBench waitlist at skillzbench.com and follow along as the benchmarking pipeline ships.