Why Your Eval Scores Lie: The Dataset Problem

Every team wants a high eval score. Far fewer ask the question that actually determines whether that score means anything: what is it being measured against? Teams obsess over metrics, judge models, and dashboards while the dataset quietly decides whether their evaluation tells the truth or flatters them.

The scale analogy from practitioner guides is hard to beat: metrics without a golden dataset are like a scale with no reference weights — you can watch the needle move, but you cannot say what is heavy. The same point shows up across the evaluation literature: the dataset matters as much as the evaluator, and a narrow or overly clean test set can produce strong scores that simply do not generalize to real traffic (Arize).

Benchmark saturation makes this worse, not better. Frontier models now cluster above 88% on MMLU and above 85% on HumanEval, so those headline numbers stopped differentiating anything years ago. When everyone scores near the ceiling, the interesting signal has moved to harder, contamination-resistant tests — and that is exactly where the dataset problem becomes visible.

Consider reported SWE-bench Pro results: GPT-5 scores 23.3% on the public benchmark, but drops to 14.9% on private subsets; Claude Opus 4.1 falls from 22.7% to 17.8%. Nothing about the models changed — only the exposure of the test cases did. That gap is data leakage in action, which practitioners call the silent killer of evaluation. If your eval scores look suspiciously high, assume leakage until you have ruled it out.

Leakage takes several forms, and not all of them involve the model having memorized the test:

• Few-shot leakage, where example cases in your prompts mirror the eval set
• Fine-tuning leakage, when training data overlaps your test cases
• Prompt-engineering-to-the-test, where you tweak prompts until the eval passes — effectively overfitting to 60 examples
• Synthetic-generation contamination, where generated test cases echo your prompts back at you

There is a practical check, too: run a near-duplicate scan across your dataset and flag pairs with a similarity ratio above 0.85. It is a five-minute script that catches the most common contamination before it catches you.

This is the philosophy behind how we are building SkillzBench. Agent skills are proliferating fast — published research suggests roughly 26% carry at least one vulnerability — and a benchmark that scores them on public, leaky, or overly clean test cases would produce Lift numbers nobody can trust. Our plan pairs a versioned, leakage-controlled benchmark corpus with per-skill with-and-without comparisons, so the score reflects the skill, not the dataset's generosity.

The uncomfortable truth: a mediocre metric on a great dataset beats a great metric on a garbage dataset every time. Before you tune anything else, audit what your eval set is actually measuring.

Want eval guidance grounded in real testing rather than vibes? Join the SkillzBench waitlist — we publish what we learn as we build.

The Golden Dataset Blueprint: Small, Real, and Well-Labeled

The most common mistake teams make when building an eval set is over-engineering it — hundreds of synthetic examples, elaborate metrics, and still no idea whether their system got better. The research points the other way: a small, real, carefully labeled dataset beats a large, fabricated one every time.

Start with 50–100 cases per feature. According to practitioner guidance on golden datasets, a focused set of 40–60 core cases plus 15–25 edge cases catches the vast majority of regressions for a single feature. Sizing is explicitly a rule of thumb, not a law — maturing products grow toward 150–300 cases, and critical or regulated flows warrant 400+.

The source of those cases matters more than the count. Every bad production answer is a free, high-value regression test case; synthetic examples are only worth generating for rare edge scenarios you haven't seen in the wild yet. Arize calls this "building evals from traces" — feeding production failures back into your dataset so it reflects real traffic rather than what you imagined users would ask.

Labeling is where golden datasets are made or broken. A case with a wrong expected output actively harms you, failing good outputs and passing bad ones — teaching you the opposite of the truth. Two practices protect against this:

• Prefer key facts over exact strings. LLM outputs are non-deterministic, so rigid expected text will fail correct answers.
• For open-ended cases, write 2–4 criterion rubrics describing expected behavior instead of a single reference answer.
• Have a second person review every label before it enters the dataset. Treat mislabeled cases like bugs — fixed via reviewed commits.

Then stress the edges. Happy-path coverage is easy and low-value; edge cases are where systems actually break. A good starting set covers out-of-scope requests, adversarial or injection attempts, ambiguous phrasing, and multi-intent inputs — roughly 20–30 edge-case scenarios to begin with. Empty or malformed inputs, sensitive topics, and long-context cases round out the picture as your set matures.

This blueprint shapes how we're building SkillzBench. Benchmarking agent skills means testing each one in a sandbox with the same model, with and without the skill, and publishing the Lift in percentage points — which only means something if the underlying corpus is small enough to re-run on every upstream change, real enough to predict production behavior, and labeled carefully enough that a wrong rubric never inflates a score. Small, real, and well-labeled isn't a compromise; it's the design.

Version It Like Code, Guard It Against Leakage

A golden dataset that lives in a spreadsheet or a shared doc dies quietly: someone edits a case, nobody knows why, and three months later your scores mean nothing. The fix is to treat your eval set exactly like source code — versioned, reviewed, and guarded against contamination.

The core practice is storing eval cases as diffable JSONL in a repository. Every change — adding a case, fixing a bad label, pruning an obsolete example — goes through a pull request with a reviewer, gets a tagged version, and lands in a changelog. This matters because a mislabeled case is a bug: it fails good outputs and passes bad ones, and as one practitioner guide puts it, a wrong expected output teaches you the opposite of the truth. Reviewed commits make label fixes auditable and reversible.

Tagging dataset versions alongside prompt and model versions is equally important. When your pass rate jumps or drops, you need to know whether the model changed, the prompt changed, or the test itself changed. Without versioned datasets, you're debugging with a broken ruler.

The second durability mechanism is leakage control — what practitioners call the silent killer of evaluation. The evidence is stark: on private held-out subsets of SWE-bench Pro, GPT-5 dropped from 23.1% to 14.9%, and Claude Opus 4.1 fell from 22.7% to 17.8% — published scores on public sets can meaningfully overstate real capability, according to benchmarking research.

Three defenses work well together:

• Keep a private held-out subset that never appears in your working set, and score against it periodically to check for drift between "practice test" and "real test" performance.
• Generate synthetic cases with a different model than the one you're evaluating — ideally a stronger one — so the test cases don't inherit the target model's blind spots.
• Run near-duplicate checks on a schedule, flagging pairs with a difflib similarity ratio above 0.85 so near-copies of your training or few-shot examples don't inflate scores.

The most insidious leakage, though, is self-inflicted. If you tweak your prompt until the eval set passes, you have now overfit to 60 examples. The scores look great; the product doesn't improve. The rule of thumb: if your eval scores are suspiciously high, assume leakage until you've ruled it out.

This is the discipline we're applying to SkillzBench's benchmark corpus. Because we re-benchmark agent skills whenever upstream changes land, the test cases themselves need the same rigor we apply to skills — pinned versions, reviewed changes, and held-out cases that measure genuine Lift rather than memorized performance. A dataset you can diff, review, and trust is the only kind that lasts.

Scoring Without Fooling Yourself: Hybrid Judging and Per-Slice Results

An 89% overall score can hide a disaster. If your easy cases flood the dataset and your judge scores everything generously, the hard cases where your system actually fails get averaged into invisibility — which is why scoring strategy matters as much as dataset construction.

The good news: you don't need humans grading every output. LLM-as-Judge has become production-viable, showing 80–90% agreement with human judgment at 500x–5000x lower cost. But judges inherit the biases of the models behind them, and those biases are measurable. In one reported test, GPT-4 showed roughly 40% inconsistency in pairwise judgments when the order of outputs was swapped, and longer responses earned about 15% score inflation purely for being verbose.

The emerging best practice is a tiered scoring pipeline that layers cheap, reliable checks under expensive, nuanced ones:

• Deterministic code checks first — exact facts, format rules, schema validation. These are free, fast, and never hallucinate.
• LLM-as-Judge scoring next, with a task-specific rubric of 2–4 criteria rather than a vague "rate this 1–10."
• Human spot audits on flagged and sampled cases, treating the judge as a first-pass filter, not the final word.
• Expert review reserved for high-stakes outputs where a wrong pass is costly.

The second discipline is per-slice scoring. Tag every case — by difficulty, category, locale, tool used — and track pass rates per slice, not just in aggregate. A single overall number lets a flood of easy wins bury a regression on your injection or ambiguous-input cases, exactly the edge cases where systems actually break. When a change ships, compare each slice against its stored baseline and fail the run on regression beyond tolerance, averaging across a few runs since model outputs vary call to call.

For agents, one more rule: evaluate the observable path and the final system state, not just the answer. A plausible-sounding response can conceal a wrong tool call, an invented parameter, or a failed action — and only the final answer is visible if that's all you measure. This is the principle behind how SkillzBench is building its skill benchmarks: every agent skill runs in a sandbox with the same model, with and without the skill, so the Lift in percentage points reflects what actually happened during execution, not just what the agent claimed.

Use judges to augment human judgment, never replace it. Score by slice. Measure the path. That combination keeps your eval set honest long after the initial build.

Make It Last: CI Regression Gating and the Maintenance Routine

A golden dataset that isn't wired into your pipeline is just a spreadsheet. The payoff comes when evaluation becomes a gate: every change runs against your stored baseline, and regressions fail the build before they reach users.

The cadence that practitioners converge on is tiered. Arize recommends running smoke evals on every pull request, full suites nightly or weekly, and online checks on canary traffic — because "model behavior can drift over time due to prompt changes, user behavior, or system updates." Set explicit thresholds per metric (for example, faithfulness at 0.8, answer relevancy at 0.7), fail on regression beyond tolerance, and track pass rate by slices so an easy-case flood can't hide a hard-case collapse.

One wrinkle: LLM outputs are non-deterministic. The same input can yield slightly different responses on each call, which means simple pass/fail testing can mislead you. The mitigation is to average metrics across multiple runs and set thresholds with a small margin, so a borderline case doesn't flicker your build green and red alternately.

Then comes the part most teams skip: the dataset itself needs a maintenance routine, because datasets rot. Practitioners suggest a recurring schedule:

• Add real production failures weekly — every bad answer is a free, high-value regression test case.
• Review labels whenever policies or product behavior change, since a wrong label teaches you the opposite of the truth.
• Prune obsolete cases that no longer reflect real traffic.
• Re-check for leakage after any prompting or fine-tuning change — if scores are suspiciously high, assume leakage until ruled out.

This is the same discipline SkillzBench is applying to agent skills. Agent skills — SKILL.md instruction packages for Claude Code, Codex, Cursor, and Gemini CLI — are constantly changing upstream, and published research suggests roughly 26% carry at least one vulnerability. That's why SkillzBench's benchmarking approach pins every skill version to a commit SHA, tests each skill in a sandbox with the same model with and without the skill to publish the Lift in percentage points, and plans to re-benchmark continuously as upstream skills change. A skill that lifted performance by 12 points in January may drag an agent down by March if its instructions drifted — and only continuous re-benchmarking catches that.

Version your dataset like code, gate your builds on it, and maintain it on a schedule. Do that, and your eval set stops being a snapshot and becomes what it should be: a durable control panel for quality. If you want that same rigor applied to the skills you install in your agents, join the SkillzBench waitlist and get the Lift data as it ships.