Agent Eval Metrics: Task Success, Cost, Latency, Safety

Why 75% of Teams Are Evaluating Agents in the Dark

Your agent just told you it completed the task. It didn't — and nothing in your evaluation pipeline will catch that. This is the reality for most teams shipping AI agents today: failures hide along multi-step execution paths where small errors compound into silent, expensive breakdowns.

The numbers are stark. A large-scale production study of 306 practitioners across 26 domains found that 75% of teams evaluate their agents without formal benchmarks, and 74% rely primarily on human judgment. A cited MIT Media Lab/NANDA study puts the failure rate for agent deployments at 95%. When nearly every deployment fails and three-quarters of teams fly blind, the two facts are not coincidental.

Why don't teams just use public benchmarks? Because they rarely transfer. The production study found public benchmarks "rarely apply" to domain-specific production tasks, and Databricks confirms that high scores on generic datasets "do not guarantee production reliability, safety or cost efficiency in real workflows." A model that aces SWE-bench can still fumble your internal invoicing workflow, because your workflow has no benchmark.

The problem runs deeper than a missing score. Agents fail in ways single-output testing can't see:

• The "ghost action" — the agent claims it completed a step without ever calling the tool, so everything appears to work while nothing actually happened
• Compounding trajectory errors, where one bad tool call snowballs into wasted tokens and poisoned context
• Cost blowouts, where two runs both "pass" while one burns far more tokens to get there

As one practitioner guide puts it, the ghost action is arguably the most dangerous failure mode precisely because everything looks fine on the surface. An end-to-end score tells you that something broke, not which component broke.

Now extend this to agent skills. Skills — SKILL.md instruction packages for Claude Code, Codex, Cursor and similar agents — are spreading fast, yet published research has found roughly 26% of skills carry at least one vulnerability. A team installing a skill from GitHub has no controlled way to know whether it improves task success, quietly inflates cost, or introduces risk. The benchmark gap that plagues agents applies doubly to the skills layered on top of them.

This is the gap SkillzBench is being built to close: an independent lab that pins every skill version, security-scans it, and runs controlled same-model, with-and-without comparisons in a sandbox — so a skill's Lift is measured, not guessed. Until teams can measure skills the way they're learning to measure agents, they're evaluating in the dark at exactly the layer where errors compound fastest.

The Four Metric Pillars: What to Actually Measure

Knowing that an agent failed is easy. Knowing which pillar broke — task success, cost, latency, or safety — requires measuring all four deliberately, because each answers a different question about the same run.

Pillar 1: Task success at multiple levels. The emerging consensus is a diagnostic stack: end-to-end outcome (did the task succeed?), trajectory (was the path efficient?), then component level (which tool broke?). An end-to-end score tells you that something broke, not which component broke, as the team at Confident AI puts it. And success must be verified against actual environment state, not agent claims. Anthropic's guidance for a flight-booking agent is blunt: the outcome is whether a reservation actually exists in the database — not what the agent said. This matters because of the "ghost action" failure mode, where an agent claims an action completed without ever calling the tool, and "everything appears to work."

Pillar 2: Cost per resolution and tokens per task. Two runs can both pass task completion while one burns far more tokens — a warning from practitioner experience that makes cost deltas a first-class metric, not an afterthought. Arize defines the core pair: cost per resolution (total agent cost divided by successful resolutions) and tokens per task. Monte Carlo frames the stakes plainly: cost per task is the difference between an AI project that grows and one that gets axed in the next budget review. A skill that lifts task success but multiplies token usage is a mixed result at best.

Pillar 3: Latency. Here the data contains a productive tension. A large-scale production study of 306 practitioners found latency blocks deployment for only 15% of applications, and 66% of organizations tolerate minutes-plus response times because agents still outperform human baselines. Yet the same body of research warns that long wait times degrade user experience and erode trust — and Arize recommends tracking P95 latency specifically, because "these are the runs users remember." Both are true: latency is rarely a blocker, but it quietly shapes how users feel about the system.

Pillar 4: Safety. This is the least-developed pillar in current practice — several major frameworks omit it entirely. Arize recommends starting with the agent's permission boundary (read-only versus write-capable agents carry very different risks), then tracking refusal rate, escalation rate (separating good escalations from bad), and unsafe tool actions — a metric that grows more important as agents move from answering questions to changing system state. The production study found security is largely manageable through action and environment constraints, which is exactly the philosophy behind SkillzBench's approach: sandboxed, same-model benchmarking of every skill with and without it, publishing the Lift in percentage points so you can see what a skill actually contributes across all four pillars.

• Outcome verification: check environment state, never agent claims
• Cost: track cost per resolution and tokens per task alongside quality
• Latency: monitor P95 even when it isn't a deployment blocker
• Safety: permission boundaries, refusals, escalations, unsafe tool actions

Hidden Failure Modes That Metrics Must Catch

The most dangerous agent failures are the ones your dashboard says nothing about. When Confident AI catalogued failure patterns from real agent deployments, the standout was the "ghost action" — an agent that claims it completed a task without ever calling the tool — which they call "arguably the most dangerous agent failure mode because everything appears to work." The transcript reads like a success story; the database, the file system, or the API tells a different one.

The pattern extends beyond ghosts. Confident AI's practitioner field guide names three more recurring culprits: the interrogation loop, where an agent burns steps asking the environment redundant questions; the confident fabricator, which invents plausible details instead of retrieving them; and the budget burner, which completes the task but multiplies token spend to get there. As Confident AI puts it, two runs can both pass task completion while one burns far more tokens.

Monte Carlo adds a compounding dimension: "bad tool calls don't stay small. They snowball." A single malformed call wastes tokens, poisons the context window, and pushes the agent further off course, degrading cost, latency, and trust simultaneously. Their evaluation guidance treats trajectory health as inseparable from outcome quality.

This is precisely why transcript-level scoring fails. A grader reading only the final response sees a fluent, confident answer and marks the run a pass. Anthropic's engineering team recommends the opposite: outcome verification against actual environment state — for a flight-booking agent, the outcome is whether a reservation exists in the database, not what the agent said. The KDD 2025 survey reinforces this, noting that completion metrics like Success Rate "provide limited fine-grained insight into failures, especially when most models achieve low success rates."

Catching these failure modes requires a two-layer approach:

• Outcome verification — check the environment, not the agent's claims, to expose ghost actions and fabrications.
• Trajectory analysis — inspect the path for loops, redundant calls, and snowballing errors that inflate cost.
• Cost-per-resolution tracking — surface budget burners that "pass" while quietly wrecking unit economics.

The stakes rise when you introduce agent skills, because a skill's entire job is to change agent behavior. This is why SkillzBench is building its benchmark pillar around sandboxed runs that verify actual outcomes and capture trajectory-level cost and latency deltas — with and without the skill, on the same model — rather than trusting what the agent reports. A skill that lifts success rates while doubling token usage is a mixed result, and evaluators need to see both halves of that trade.

Controlled Comparison: How Lift Measurement Answers the Benchmark Gap

The benchmark problem in agent evaluation is well documented: 75% of production teams evaluate without formal benchmarks because public ones rarely apply to their domain-specific tasks. A large-scale study of 306 practitioners found teams deliberately build custom, workload-specific tests instead — and Databricks confirms that custom benchmarks consistently outperform generic datasets. The gap, then, isn't methodology. It's that nobody has applied controlled comparison to agent skills.

That's the premise behind SkillzBench's Lift measurement. Instead of asking "is this skill good?" in the abstract, it asks a narrower, more answerable question: does this specific skill version improve task outcomes for a specific model, and at what cost?

The design follows directly from the research. Every skill version is pinned to a commit SHA, so results can't silently drift when upstream files change. Each skill runs in a sandbox against the identical model twice — with the skill and without — and the difference in task success is published as Lift, in percentage points, alongside cost and latency deltas. That last part matters because two runs can both pass task completion while one burns far more tokens; a skill that adds 10 points of success while doubling token usage is a mixed result, not a win.

The differentiator is outcome-based verification. Anthropic's guidance on agent evals is that success should be measured by actual environment state — for a flight-booking agent, whether a reservation exists in the database, not what the agent said. This guards against the failure mode practitioners dread most: the "ghost action," where an agent claims a step completed without calling the tool and everything appears to work. SkillzBench's benchmark design verifies what actually happened in the sandbox, not what the transcript claims.

Just as important is what Lift is not. It is published separately from the Trust Score, which draws only on security, quality, license, maintenance, and community signals. Money never touches scores — a skill's rating can't be influenced by its benchmark results or by anything its author pays. A skill can be safe and well-maintained but deliver no Lift, or deliver strong Lift while carrying security concerns; showing both independently lets users weigh that trade-off themselves.

This approach mirrors how the production study found teams actually manage agent risk — through action and environment constraints rather than blind trust. Sandboxed, same-model, with-and-without testing is simply that discipline applied to skills, with results published in percentage points anyone can read.

SkillzBench is building this now, and the waitlist is open for teams who want skills, proven.

Make Evaluation Continuous: Drift, Re-Scans, and Your Next Steps

A benchmark you ran once is a snapshot of an agent that no longer exists. Monte Carlo puts it bluntly: agents in production "break, drift, and change without bothering to warn you," and metrics are "pretty much useless if you only glance at them once a quarter in a Jupyter notebook" (Monte Carlo's evaluation guide). Arize agrees, warning that "fixed metrics fail when the agent outgrows the original failure mode" (Arize's metrics framework).

The same logic applies to the skills your agents depend on. A SKILL.md package pinned today can change upstream tomorrow — and with published research finding roughly 26% of skills carry at least one vulnerability, an untracked update is a security event, not a housekeeping item. This is why SkillzBench is building its Stay Current pillar to track upstream changes, re-scan for vulnerabilities, and re-benchmark so a skill's Lift and Trust Score reflect what the skill is now, not what it was at install time.

Re-benchmarking matters for another reason: the production study found 75% of teams evaluate without formal benchmarks at all (a large-scale study of 306 practitioners), and public benchmarks "rarely apply" to domain-specific tasks. A skill that lifted your agent's success rate last quarter may be neutral — or harmful — after a model upgrade. Anthropic notes that teams with evals can adopt new models "in days" while teams without face weeks of testing (Anthropic's eval guidance); continuous re-benchmarking extends that speed to your skill stack.

Wherever you land on tooling, build these habits into your routine:

• Verify environment state, not claims. Anthropic's outcome-based approach — check whether the reservation exists in the database, not what the agent said — catches the "ghost action" failure where everything appears to work.
• Track cost per resolution. Total agent cost divided by successful resolutions, per Arize's framing, surfaces skills that pass tasks while burning tokens.
• Define permission boundaries per skill. Read-only and write-capable agents carry very different risk profiles; match each skill's permissions to its actual job.
• Re-scan on every upstream change. Treat a skill update the way you'd treat a dependency bump in production code.

If you'd rather not build this pipeline yourself, join the SkillzBench waitlist at skillzbench.com for early access to verified, benchmarked skills — pinned to commit SHAs, security-scanned, and re-benchmarked as they change. Skills, proven.