How to Compare LLMs Fairly: A Buyer’s Method

Why Leaderboard Scores Keep Lying to Buyers

Every buyer has experienced it: a model tops the leaderboard, wins the internal debate, ships to production — and then underperforms a cheaper alternative on the actual workload. The problem isn't your team's judgment. The leaderboard itself is structurally unreliable, and three forces explain why.

Contamination is the first culprit. When models train on massive web corpora like Common Crawl and The Pile, benchmark contamination is not a possibility but an inevitability — the only question is the degree. A peer-reviewed systematic review of 55 studies found contamination inflates scores by roughly 6%–40%, with documented evidence in widely cited benchmarks like MMLU, GSM8K, and HumanEval. Worse, no detection method works reliably across all contamination tiers, and instruction tuning remains a persistent blind spot.

Saturation compounds the problem. A February 2026 saturation analysis found high or very high saturation across nearly half of reviewed benchmarks. Legacy benchmarks like GLUE, SuperGLUE, MMLU, and GSM8K no longer separate frontier models at all. As the Stanford AI Index puts it, evaluations intended to challenge models for years are saturated in months. A score that once signaled excellence now signals little more than "this model has seen the test."

The third issue is the gap between benchmark and production performance, which isn't a calibration issue — it's structural. Benchmarks use clean, standardized data; production data is messy and domain-specific. The consequences show up in three predictable ways:

• A model that ranks first on MMLU can lose to a cheaper model on your document-summarization workload.
• A model scoring 95% at 4,000 tokens per task can have worse cost-per-outcome than an 88% model averaging 1,800 tokens.
• An 800ms median benchmark latency can become 2,400ms at P95 under real production concurrency.

None of this means benchmarks are useless — they're a reasonable screening filter. But they can't answer the only question that matters to a buyer: does this model win on my workload, under my conditions? That requires controlled, with-and-without testing on the actual task, which is exactly the methodology SkillzBench is building for agent skills — measuring Lift in percentage points under identical conditions rather than trusting static scores. Published research has already found that roughly 26% of skills carry at least one vulnerability, so the cost of skipping that validation step is real. The headline number gets you a shortlist; only your own workload gets you a decision.

The Buyer's Method: Screen, Validate, Monitor

If you've ever picked a model off a leaderboard only to watch it stumble on your actual workload, you already know the problem: public scores don't predict production results. The fix isn't a better leaderboard — it's a staged method that treats benchmarks as one input among several.

The buyer's method has three stages: screen, validate, monitor. Each stage uses a different kind of evidence, and none of them alone is sufficient.

Stage 1: Screen with discriminating public benchmarks. Use public scores only to build a shortlist. Be selective about which benchmarks you trust — a February 2026 saturation analysis found high or very high saturation across nearly half of benchmarks reviewed, and legacy standards like MMLU, GSM8K, and HumanEval no longer separate frontier models. Newer, harder benchmarks (HLE, BBEH, FrontierMath, ARC-AGI-2/3) still discriminate, so prefer those for screening.

Stage 2: Validate with application-specific evals on real data. Benchmarks are simply "not suitable for evaluating LLM-based products," as Evidently AI puts it — your product needs custom datasets and criteria. Build a small evaluation set of 100–500 representative production examples plus a 20–50 case regression set, run each prompt 3–5 times to capture variance, and load-test at roughly twice your expected peak concurrency. Measure four dimensions: output quality (blind, to counter model-identity bias), cost per completed outcome, latency at P95/P99, and consistency. Cost per outcome matters more than raw accuracy — a model scoring 95% while averaging 4,000 tokens per task can cost more per useful result than an 88% model averaging 1,800.

Discount headline scores for contamination, too. A peer-reviewed systematic review of 55 studies estimated benchmark inflation of roughly 6%–40% depending on the benchmark, and found no detection method reliably catches it across all tiers. Ask vendors directly whether they've run contamination analysis — disclosure practices established by the GPT-3, OPT, and PaLM teams set the standard buyers should demand.

Stage 3: Monitor continuously. Providers change model behavior without new version numbers, and benchmarks themselves decay through contamination and saturation. One-time evaluation goes stale; production benchmarking has to be continuous.

This staged philosophy — same conditions, with-and-without comparison, results published separately from any rating — is exactly what SkillzBench is building for agent skills. Its planned Lift metric tests every skill in a sandbox with the same model, with and without the skill installed, so the number reflects the skill's actual contribution rather than a static score. And because skills change upstream, the "stay current" pillar re-scans and re-benchmarks as components evolve.

• Screen with still-discriminating benchmarks to build a shortlist — never to make the final call.
• Validate on 100–500 real production examples, measuring quality, cost per outcome, latency, and reliability.
• Discount public scores for contamination risk and ask vendors for disclosure.
• Monitor continuously, because models change silently and benchmarks decay.

The method isn't complicated, but it demands discipline: no single number, no one-time test, and no trust in scores you can't reproduce. That discipline is what separates a fair comparison from an expensive guess.

Compare on Four Axes — Never a Single Score

A single number can make two wildly different models look identical. That's why fair comparison demands four separate axes — quality, cost, latency, and consistency — each measured on your workload, not a leaderboard's.

Start with task-specific output quality, evaluated blind. Raters who know which model produced an output bring brand bias along with their judgment, so strip model names before scoring. Vague criteria like "good output" produce evaluations that aren't reproducible and can't justify a vendor decision, according to enterprise evaluation guidance. Build a set of 100–500 recent production examples, plus a 20–50 case regression set for edge cases — that's where the gap between "95% on benchmark" and "broken in production" lives.

Second, measure cost per outcome, never cost per token. Token prices hide the fact that verbose models burn budget on the way to an answer. A model scoring 95% while averaging 4,000 tokens per task can cost more per completed outcome than an 88% model averaging 1,800 tokens — the cheaper model wins on the metric that reaches your invoice. This is the same discipline SkillzBench applies when benchmarking agent skills: Lift in percentage points is always reported separately from trust scores, because capability and cost are different questions that deserve different numbers.

Third, test latency under real concurrency, not vendor medians. A model with an 800ms median benchmark latency can deliver 2,400ms at P95 under production traffic. Track the percentiles that users actually feel:

• P95 and P99 latency, load-tested at roughly 2× expected peak concurrency
• Time to first token (TTFT), which dominates perceived responsiveness in streaming interfaces
• Run-to-run variance — 3–5 runs per prompt to expose inconsistency

Fourth, demand consistency. A model that nails a task four times out of five is a different product than one that nails it every time, even at identical average scores. Cache behavior matters too: a 35% cache hit rate can flip an entire model comparison.

The through-line: never collapse four axes into one score. Intelligence and price aren't correlated across the market — top-intelligence models can cost roughly 100x more than low-cost ones for a modest capability spread, as multi-metric leaderboards show. A buyer who sees only a composite number sees none of the trade-offs.

Build Your Own Eval Kit: A Practical Checklist

Vague criteria like "good output" produce evaluations that can't justify a vendor decision — so your eval kit needs to be concrete enough to defend in a budget meeting. The good news: a credible kit is smaller and cheaper than most teams assume.

Start with the dataset. Enterprise evaluation guidance recommends 100–500 representative production examples plus a 20–50 case regression set covering known edge cases. Have domain experts review every example, and use blind evaluation so reviewers can't tell which model produced which output — model-identity bias is real.

Then run it properly:

• Run each prompt 3–5 times to capture variance, not just a single lucky sample.
• Load-test at roughly 2× your expected peak concurrency — a model with an 800ms median benchmark latency can deliver 2,400ms at P95 under production load.
• Track cost per completed outcome, not per token. A 95% model averaging 4,000 tokens per task can cost more per outcome than an 88% model at 1,800.
• Log P50/P95/P99 latency and time-to-first-token separately, since benchmarks rarely transfer.

Next, pressure-test the scores you're discounting. Contamination-driven inflation is estimated at roughly 6%–40% depending on the benchmark, and a peer-reviewed review of 55 studies found no detection method is consistently reliable across contamination tiers. So demand transparency from vendors. The GEM 2026 Outstanding Paper proposes a Contamination Transparency Card for benchmark releases — treat it as a checklist you ask vendors to complete. Disclosure precedents exist: the GPT-3 paper flagged evaluation sets with significant overlap, and OPT and PaLM extended that analysis as standard practice.

Finally, make the kit permanent rather than one-shot. Providers update model behavior without new version numbers, so production benchmarking has to be continuous — re-run your regression set whenever a vendor ships a change, and track drift over time.

That continuous-retest principle is exactly what SkillzBench is building into its methodology: re-scanning and re-benchmarking agent skills as upstream components change, and measuring Lift with-and-without under identical conditions rather than trusting static scores. If you're evaluating skills alongside models, join the waitlist — the same buyer's skepticism applies to both.

The Same Logic Applies to Agent Skills

Once you've learned to compare models fairly, a new problem appears: the agent skills layered on top of them. A skill that promises to make your coding agent better at code review might help, might do nothing, or might quietly introduce a security risk — and most teams have no way to tell which.

The same buyer's method applies, almost unchanged. The core principle is with-and-without testing under identical conditions: run the same task in the same sandbox with the same model, once with the skill loaded and once without, and measure the difference. Anything less invites the same contamination-style inflation that plagues LLM benchmarks, where peer-reviewed estimates put performance inflation at roughly 6%–40% depending on the benchmark. A skill author claiming "my skill improves results by 30%" without a controlled baseline is making the same mistake leaderboard shoppers make.

This is exactly the methodology SkillzBench is building into its Lift benchmark: every skill tested in a sandbox with the same model, with and without the skill, so the published Lift in percentage points reflects the skill's actual contribution rather than the model's raw capability. Just as independent model leaderboards separate intelligence from price, Lift is always shown separately from trust scores — capability and safety are different questions, and conflating them serves nobody.

Fair comparison at the skill layer also demands rigor on three fronts that model evaluation doesn't require:

• Security scanning — published research found roughly 26% of agent skills carry at least one vulnerability, so any benchmark that ignores safety is incomplete by design.
• Version pinning — every skill version pinned to a commit SHA, because "skill X performed well" is meaningless if the skill changes underneath the result.
• Continuous re-benchmarking — providers update models without new version numbers, and the same drift happens upstream in skills, so evaluation has to be continuous, not a one-time exercise.

That last point deserves emphasis. A skill's Lift isn't a permanent property — it's a measurement of a specific skill version running against a specific model version on specific tasks. When either side changes, the measurement can change. SkillzBench's "stay current" pillar exists precisely for this: track upstream changes, re-scan, re-benchmark, and notify when a skill you rely on has shifted.

The buyer's takeaway is simple. Ask any skill author or marketplace the same questions you'd ask a model vendor: What was the baseline? Was the model held constant? What version was tested? When was it last re-validated? If the answer to any of those is silence, you're not looking at a measurement — you're looking at a claim. Skills, proven — that's the standard worth waiting for.