Why Traditional Metrics Can't Grade AI Agents

For years, AI teams graded language models with the same yardsticks used for machine translation: count the word overlap against a reference answer, compute a score, ship. That approach collapses the moment your model produces open-ended output — which is exactly what modern AI agents do all day long.

BLEU and ROUGE compare word overlap against reference answers, and they fall apart for open-ended generation, where many phrasings are equally valid. The G-Eval research (Liu et al., EMNLP 2023) quantified just how badly: on the Topical-Chat dataset, G-EVAL-4 achieved a 0.588 Spearman correlation with human judgments, while ROUGE-L managed just 0.244 and BLEU-4 0.259. As practitioners point out, optimizing for a metric that doesn't match human judgment actively misleads your development process.

If automated metrics fail, why not just use people? Because production systems process millions of requests, and as Splunk's analysis puts it, human-only evaluation is mathematically impossible at that volume — matching a single LLM judge overnight would require 50+ full-time annotators.

Worse, human review isn't a neutral gold standard. Research shows human evaluators rate assertive-but-incorrect outputs 15-20% higher than accurate-but-cautious ones, penalizing honest epistemic markers by 0.7 points on a 5-point scale despite factual equivalence. Confidence reads as competence, even when it shouldn't.

Agent skills — the SKILL.md instruction packages that steer Claude Code, Codex, Cursor and other agents — make this problem sharper. A skill's value shows up in judgment-dependent dimensions: planning, tool selection, and multi-step behavior, which evaluation research identifies as distinct judgeable aspects of agent performance. No word-overlap metric can tell you whether a skill helped an agent reason better.

The stakes are real: published research has found roughly 26% of skills carry at least one vulnerability. Teams need a scalable, systematic quality signal — the gap SkillzBench is building its verification and benchmarking pillars to fill, testing every skill in a sandbox with and without it to measure actual Lift.

The evaluation problem, in short:

• Deterministic metrics correlate poorly with human judgment on open-ended tasks
• Human review can't scale to production volume — and humans carry their own biases
• Agent skills demand judgment-based evaluation across planning, tool use, and reasoning

Evaluating is easier than generating — that asymmetry is the insight that makes a scalable answer possible.

How LLM-as-a-Judge Works: Three Architectures

The counterintuitive insight behind LLM-as-a-judge is that evaluating an output is easier than generating one. The judge performs a simpler, focused classification task rather than open-ended creation, which is why the technique works at all — as Evidently AI explains, you can tune LLM judges to match human judgments precisely because verification demands less than generation.

The technique was formalized in Zheng et al.'s 2023 NeurIPS paper introducing MT-Bench and Chatbot Arena, where GPT-4 as judge achieved over 80% agreement with crowdsourced human preferences — roughly matching human-to-human agreement levels. From that work and subsequent research, three judge architectures have become standard.

• Pairwise comparison — the judge picks the better of two responses. Easier than absolute scoring, but it scales quadratically: 5 variants means 10 comparisons, 10 variants means 45.
• Pointwise scoring — the judge rates a single output against criteria, reference-free or anchored to a gold standard. This is the most common production setup because it can run on every trace.
• G-Eval (chain-of-thought judging) — from Liu et al. (EMNLP 2023), the judge first generates evaluation steps, then scores using probability-weighted token logprobs for better calibration.

G-Eval's numbers explain its appeal: on Topical-Chat, G-EVAL-4 achieved a 0.588 Spearman correlation with human judgments, versus 0.273 for BERTScore and 0.259 for BLEU-4, according to Galileo's analysis. But chain-of-thought isn't a free lunch — removing CoT steps only drops correlation from 0.514 to 0.500, and follow-up empirical work found auto-CoT doesn't always improve alignment with human ratings. The cost is real too: G-Eval needs at least two API calls per dimension, so four dimensions across 100,000 daily interactions means 800,000+ calls per day.

Each architecture has a natural home. Pairwise comparison suits periodic A/B decisions rather than continuous monitoring, because of that quadratic scaling. Pointwise scoring fits ongoing monitoring where you need a score on every output. This maps directly onto how SkillzBench is designing its benchmarking: pairwise comparison is the natural fit for measuring Lift — running the same model with and without a skill in a sandbox, with order-swapped comparisons to control for position bias — while pointwise judging supports the continuous re-scoring as skills and models change.

The choice matters less than the discipline around it. Arize's guidance is blunt: binary pass/fail outputs tend to produce more stable and reliable evaluations than subtle numeric scoring, a finding echoed by Hamel Husain's work across 30+ companies, where domain-expert pass/fail judgments correlated better with actual quality than granular scores.

The Biases Every Judge Has — and How to Defuse Them

A judge that flips its verdict when you swap two answers isn't an evaluator — it's a coin with better vocabulary. Yet that's exactly what researchers found with GPT-4: IJCNLP 2025 research showed that reversing the order of responses can flip the model's judgment entirely. Position bias is the best-known failure mode, but it has siblings.

Verbosity bias rewards longer answers even when the extra length is pure padding. Self-preference bias is subtler — models rate outputs from their own family higher, and NeurIPS 2024 research found a linear correlation between a model's ability to recognize its own output and the strength of its self-preference. Finally, leniency clustering: scores pile up in the middle of the scale, making a "7" nearly meaningless.

The mitigations are proven and practical:

• Run every pairwise comparison twice with the order swapped, then average — the standard fix for position bias.
• Use a judge from a different model family than the system under test to neutralize self-preference.
• Deploy an LLM jury — multi-judge consensus has achieved Cohen's Kappa around 0.95 in published research.
• Score binary pass/fail instead of numeric scales — across 30+ companies, pass/fail judgments correlated better with actual quality.
• Include conciseness in the rubric and calibrate against human expert labels with a target of Cohen's κ > 0.8.

That calibration target matters more than any single technique. A judge that agrees with human experts at κ > 0.8 is trustworthy infrastructure; below that, you're optimizing against noise. As Arize's evaluation guide puts it, repeating the same decision doesn't establish that the decision is correct.

This is why SkillzBench is designing its benchmarking pipeline around these defenses from day one — order-swapped comparisons for measuring Lift, external-family judges, and calibration against human labels before any score informs a published result. A judge is only as good as its bias controls.

Building a Trustworthy Evaluation Stack: Layered, Not Judge-Only

No serious team uses an LLM judge as its only evaluator — the consensus architecture is layered, with each layer doing what it does best. As Splunk's practitioners put it, "the real question isn't 'LLM or human,' it's knowing when each method is the right tool, and how to architect around the weaknesses of both."

The first layer is deterministic. Structural checks — valid JSON, format compliance, safety and policy scans — are simply better handled by code than by a judge, and Arize's guidance is explicit: use code-based evaluators for deterministic checks, LLM judges only where evaluation requires interpreting meaning.

The second layer is semantic. LLM judges handle the bulk of quality assessment that deterministic rules can't touch, with binary pass/fail judgments preferred over granular numeric scores for stability — Hamel Husain's work across 30+ companies found pass/fail judgments correlated better with actual quality than numeric scales.

The third layer is human. Periodic expert calibration anchors the whole stack, and properly calibrated judges combined with human validation achieve over 80% agreement with human preferences. A typical layered stack looks like:

• Deterministic checks first: format, safety, security scans
• LLM judges for semantic quality at volume
• Periodic human expert review to calibrate and catch criteria gaps

The discipline matters as much as the architecture. Elite teams allocate 15-25% of engineering time to evaluation development, and 92% of teams integrate evaluations into CI/CD pipelines, re-running them continuously rather than treating evals as one-time setup. Teams that write evals early and run continuously "catch regressions before they compound."

This layering maps directly to how SkillzBench is building skill verification. Deterministic layers handle what should never be subjective: pinning every skill version to a commit SHA, license checks, and security scans — critical given published research finding roughly 26% of skills carry at least one vulnerability. LLM judges then assess semantic quality, and the Benchmark pillar tests every skill in a sandbox with the same model, with and without the skill, publishing the Lift in percentage points — shown separately from trust scores, so a skill that helps and a skill that's safe are never conflated. The Stay-current pillar closes the loop: when upstream changes, re-scan, re-benchmark, notify.

The result is a stack where each layer compensates for the others' weaknesses — trustworthy evaluation comes from the combination, not from any single judge.

Putting It to Work: Judging Agent Skills with Lift

Knowing the biases is one thing; building an evaluation pipeline that neutralizes them is another. The good news is that the research points to a clear recipe for measuring whether an agent skill actually helps.

The core design is a controlled A/B: run the same model, on the same task, once with the skill loaded and once without. Pairwise comparison fits this naturally — judges find "which response is better" easier than absolute scoring, because verifying an answer is easier than generating one. The comparison produces a Lift figure: how much the skill improves output, in percentage points.

But pairwise judging has a known weakness. Position bias is real — IJCNLP 2025 research found that simply swapping answer order can flip a judge's verdict. The fix is to run every comparison twice with the order swapped and average the results, so a skill's Lift reflects genuine quality rather than where its output happened to appear.

A few practices separate trustworthy Lift numbers from noisy ones:

• Prefer binary judgments — Arize practitioners report that binary outputs produce more stable evaluations than subtle numeric scoring, and pass/fail correlates better with actual quality.
• Keep the judge external to the model family under test. Self-preference bias means models rate their own family's outputs higher.
• Layer deterministic checks first — format, safety, and structural validation — before semantic judging.
• Re-benchmark when upstream changes occur, since a skill's Lift can shift with every model or dependency update.

This is exactly the architecture SkillzBench is building into its pre-launch Verify and Benchmark pillars: every skill version pinned to a commit SHA, validated and security-scanned deterministically, then tested in a sandbox with the same model with and without the skill. One design principle matters most: Lift is always shown separately from ratings. A skill can be well-written and still not help; a rough-looking one might deliver real gains. Collapsing those into a single score hides exactly what users need to know.

The field's own guidance reinforces this discipline. A peer-reviewed survey in The Innovation concludes that judge reliability "remains a significant challenge requiring careful design and standardization" — which is why treating benchmarking as continuous infrastructure, not a one-time report, is the only approach that holds up.

Skills, proven. SkillzBench is in active pre-launch development — join the waitlist at skillzbench.com to follow the build and get early access to independent skill benchmarks.