Why Agent Evaluation Breaks Everything You Know About Testing

Every testing instinct you've developed breaks down the moment software starts making its own decisions. An LLM produces text; an agent takes actions — and the evaluation method that works for one fails for the other, as Toloka's analysis of agent benchmarks puts it plainly.

The distinction isn't academic. A KDD 2025 survey from SAP Labs researchers offers a useful analogy: LLM evaluation is like examining an engine's performance, while agent evaluation assesses a car's performance comprehensively under various driving conditions. The engine can be perfect and the car can still crash. Lilian Weng frames agents as "decision-making entities that use perception, planning, and action to solve tasks over time" — which means you're no longer grading an output, you're grading a trajectory through a world that changes as a result of the agent's actions.

That breaks both testing traditions at once:

• Traditional software testing assumes deterministic behavior — same input, same output. Agents are stochastic and non-deterministic, so identical runs can diverge.
• LLM evals check text quality. Agents change state — they book flights, write files, call tools — so the transcript can look flawless while nothing actually happened.
• Multi-step behavior compounds errors. A 95%-accurate step repeated ten times is not a 95% task.

The consequences show up in production data. A large-scale study of 306 practitioners found that 95% of agent deployments fail, with reliability the top unsolved challenge — and evaluation difficulty directly limits teams' ability to verify correctness. The same study found 75% of teams evaluate agents without formal benchmarks, because production tasks are too domain-specific for public suites. Anthropic's engineering team adds a subtler trap: their Opus 4.5 "failed" a flight-booking eval by discovering a policy loophole that actually produced a better outcome than the test anticipated.

This is the gap that independent verification has to fill. Agent skills — the instruction packages steering agents in tools like Claude Code and Cursor — are proliferating faster than anyone can vet them, and published research suggests roughly 26% carry at least one vulnerability. It's why SkillzBench is building controlled benchmarking: test every skill in a sandbox with the same model, with and without the skill, and publish the difference in percentage points as Lift. A/B comparison under fixed conditions is the only way to isolate what a skill actually contributes.

The old playbooks can't tell you whether an agent — or the skill steering it — actually works. The rest of this article covers the metrics that can.

The Metrics That Actually Matter: Outcomes, Not Transcripts

The scariest agent failure is the one you never see. Anthropic's engineering team draws a sharp line between the transcript — what the agent says — and the outcome, the actual final state of the world. Their canonical example: a flight-booking agent that cheerfully reports "Your flight has been booked" while no reservation exists in the database (Anthropic's evaluation guide).

Confident AI names this pattern the ghost action, calling it "arguably the most dangerous agent failure mode because everything appears to work" (Confident AI's evaluation guide). Everything checks out on the surface — the tone is confident, the summary is polished, the user walks away satisfied — until reality catches up. This is why outcome-based grading, checking the environment itself rather than the agent's account of it, is becoming the gold standard.

The ghost action has siblings worth knowing by name:

• The interrogation loop — the agent repeatedly asks for information the user already provided.
• The confident fabricator — polished output built on stale tool data.
• The budget burner — the task technically completes, at wildly excessive cost.

That last pattern points to a bigger problem: task completion alone tells you surprisingly little. A KDD 2025 survey notes that success rate provides "limited fine-grained insight into failures, especially when most models achieve low success rates" (the survey). An agent can pass the task while failing on economics and feel.

So a serious metric stack reaches well beyond a pass/fail checkbox. It covers tool-call precision — wrong function names, missing parameters, hallucinated arguments (IBM's evaluation framework). It tracks cost in tokens and tool calls, latency, steps to completion, and run-to-run variance, since trajectory metrics need distributions rather than averages to expose long-tail problems (Toloka's benchmark analysis). And it measures safety and UX as independent dimensions, because an agent can be safe under adversarial testing yet drift into unsafe patterns in normal operation.

This multidimensional view is exactly what shapes how we at SkillzBench are building our benchmarking approach: testing every skill in a sandbox with the same model, with and without the skill, and publishing the Lift as a percentage point difference — an outcome-based, controlled comparison rather than a vibes-based rating. Performance stays separate from trust, because the research is clear they're different questions.

The takeaway for any team shipping agents: grade the world the agent left behind, then grade how much it cost to get there and how it felt along the way.

ctaText: See how SkillzBench measures real Lift — join the waitlist at skillzbench.com socialProofText: Skills, proven. SkillzBench is building independent, outcome-based benchmarks for AI agent skills.

What Production Teams Actually Do (The Data)

The gap between benchmark leaderboards and production reality is wider than most evaluation guides admit. The best evidence comes from a large-scale MAP study of 306 practitioners — spanning UC Berkeley, Stanford, and IBM Research — who analyzed 86 deployed or piloted agent systems to document what teams actually do, not what conference talks recommend.

The headline finding: 74% of deployed agents rely primarily on human-in-the-loop evaluation, and 52% use LLM-as-a-judge — but every single team using an LLM judge also layers human verification on top. No production team in the study trusts a model-based grader alone. Model judges are treated as a throughput multiplier for human review, never a replacement for it.

The benchmark numbers are even more striking. 75% of teams skip formal benchmarks entirely, not because they're unaware of them, but because production tasks are too domain-specific for SWE-bench or τ-Bench to predict anything meaningful about their agents. Toloka identifies the structural reasons: contamination, task mismatch, and saturation make public suites unreliable proxies for real-world performance.

So why do benchmarks still dominate the conversation? The answer is that they serve a different purpose. Anthropic's engineering team describes the three grader types practitioners actually combine:

• Code-based graders — fast, cheap, objective outcome checks, but brittle when tasks have many valid solutions
• Model-based graders — flexible LLM-as-a-judge scoring, but non-deterministic and requiring human calibration
• Human graders — the gold standard, but expensive enough that teams reserve them for sampled traces and edge cases

Hybrid grading dominates because each grader type fails differently. Code checks verify outcomes like "the reservation exists in the database" but can't judge nuance; LLM judges handle nuance but hallucinate scores; humans catch everything but don't scale. The MAP data shows production teams stitch all three together pragmatically.

This reconciles the apparent contradiction. Public benchmarks like SWE-bench — where solve rates climbed from 40% to over 80% in roughly a year, per Anthropic — are excellent for tracking capability progress across the field. But a shipping decision ("does this agent work for our support workflow?") requires a controlled, custom comparison: the same task, the same model, run with and without the component you're evaluating.

That with/without, same-model A/B structure is exactly what SkillzBench is building for agent skills — testing each skill in a sandbox against identical tasks, so the Lift you see reflects the skill itself, not benchmark noise. It's the evaluation pattern production teams already trust, applied to the layer nobody has measured yet.

Applying This to Agent Skills: The Lift Method

Every principle above — outcome-based grading, controlled comparison, continuous re-testing — applies directly to a niche no published research covers: evaluating SKILL.md instruction packages that extend agents like Claude Code, Codex, Cursor, and Gemini CLI. A skill is not an agent; it's a capability you bolt onto one. That distinction changes the question from "is this agent good?" to something sharper: does this skill make the same agent measurably better?

The answer requires a controlled experiment, not a vibe check. That's the Lift method SkillzBench is building: run the same model in a sandbox on the same tasks, twice — once with the skill loaded, once without — and publish the difference in percentage points. This mirrors exactly how production teams already evaluate: A/B testing is a named evaluation method in large-scale practitioner research, and Anthropic's engineering team insists outcomes be graded on final environment state, not on what the agent says it did. A skill that produces confident, polished instructions but doesn't move task outcomes is the instruction-package equivalent of what Confident AI calls "the ghost action" — everything appears to work.

Lift alone, though, is only half the picture. The research consistently separates capability from safety and cost as distinct pillars, and warns that agents can pass task completion while failing on economics or security. That's why SkillzBench will always show Lift separately from its trust score, which draws on security, quality, license, maintenance, and community signals. A skill can lift performance and still be a liability.

• Capability (Lift) answers "does it help?" — measured in a sandbox, same model, with and without the skill.
• Trust answers "is it safe to load?" — security scanning, license checks, quality judging, maintenance and community signals.
• Neither can substitute for the other — a high-Lift skill with a vulnerability is worse than a mediocre skill with none.

The security half is not theoretical. Published research has found roughly 26% of agent skills carry at least one vulnerability, and Toloka's evaluation framework notes that an agent can be safe under adversarial testing yet drift into unsafe patterns under normal operation — which argues for repeated scanning, not one-time checks. SkillzBench's design accounts for this: skills are pinned to a commit SHA, scanned and benchmarked on arrival, then re-scanned and re-benchmarked whenever upstream changes appear, because continuous evaluation, not a static snapshot, is where the field has landed.

SkillzBench is pre-launch and building exactly this — initially for instruction-only skills (SKILL.md plus text references), with script-bearing skills listed link-out only. The goal is simple: a number you can trust, published separately from a score you can verify, so no one has to guess whether a skill actually helps before loading it into a working agent.

Your Action Plan: Building Continuous, Trustworthy Evaluation

Most teams know their agent evals are weak — they just don't know where to start fixing them. The good news is that expert guidance converges on a short list of practices that turn evaluation from a checkbox into a competitive advantage.

Start by preferring programmatic checks over LLM judges wherever possible. Anthropic's engineering team categorizes code-based graders as fast, cheap, and objective, and Arthur AI's advice is blunt: don't force an LLM eval where a programmatic check would do. When you do use model-based judgment, calibrate it against human review — the production research shows every team using LLM-as-a-judge also verifies with humans, never the judge alone.

Second, keep scoring simple and legible:

• Use binary pass/fail scoring — LLMs are inconsistent range scorers, returning "a 4 on one run and a 6 on another."
• Keep every eval describable in a single sentence; if you can't, it's too broad.
• Look at distributions, not means — averages hide the long-tail failures that destroy user trust.
• Re-run evals on every meaningful change, not just at launch.

That last point is the one most teams miss. Toloka describes the emerging standard as "living evaluation infrastructure" — re-evaluated continuously rather than run once and archived. Arthur AI's argument for why is compelling: user reports are a lagging signal, and because "user trust in AI agents is already fragile, a bad experience at the wrong moment can permanently change how someone thinks about your product."

This is exactly the philosophy SkillzBench is being built around. Every skill version gets pinned to a commit SHA, so a benchmark result always refers to a specific, reproducible artifact. When an upstream skill changes, the plan is to re-scan and re-benchmark it, then notify users — evaluation as ongoing monitoring, not a one-time stamp of approval. And because Lift (with-and-without performance) stays separate from the trust score, you always know both whether a skill helps and whether it's safe to use.

The stakes justify the rigor. A large multi-institution study of deployed agents found reliability remains the top unsolved challenge, and one cited report puts agent deployment failure rates at 95%. Continuous evaluation catches failures before users do — and before fragile trust is damaged beyond repair. Treat your evals like the systems they protect: always on, always current, and never trusted blindly.