The Economics of Running Agent Benchmarks

The Hidden Cost of Not Benchmarking Agent Skills

Every AI agent skill you install is a small act of trust — and the numbers say roughly one in four of those acts is misplaced. Independent academic research analyzing 31,132 skills found that 26.1% carried at least one vulnerability, spanning prompt injection, data exfiltration, privilege escalation, and supply chain risks. Snyk's ToxicSkills research puts the figure even higher, at 36.82% of 3,984 skills found flawed.

The consequences are not hypothetical. The ClawHavoc campaign documented in OWASP's Agentic Skills Top 10 case studies pushed 1,184 malicious skills through 12 compromised publisher accounts, reaching 247,693 confirmed installations. The bill: $2.3M in stolen cryptocurrency plus $890K in cleanup costs — and at its peak, 5 of the top 7 most-downloaded skills on one platform were malicious. One skill needed just three lines of markdown to exfiltrate SSH keys.

Security is only half the cost of skipping measurement. The other half is invisible: you cannot tell a skill that helps from one that hurts.

• Curated skills raise average pass rates by 16.2 percentage points — a meaningful gain, per SkillsBench research.
• But the range is enormous: +51.9pp in Healthcare versus just +4.5pp in Software Engineering.
• And 16 of 84 benchmarked tasks showed negative deltas — the skill made the agent worse.

That variance is the real problem. Adopting skills without measurement means a +51.9pp skill and a negative-delta skill look identical in a README. Both have a nice description, a GitHub repo, and a plausible use case. Only a controlled test — same model, same task, with and without the skill — reveals the difference. As the SkillsBench authors note, without this kind of evaluation, "practitioners cannot make informed decisions about Skills adoption."

Verification and benchmarking cost time and effort, but the alternative is already priced: seven-figure incident losses, security reviews after the breach, and agents quietly underperforming because a skill is bloated, misleading, or actively harmful. Benchmarking is cheaper than incident response — every time.

This is exactly the gap SkillzBench is being built to close: an independent, free directory where every skill is security-scanned, pinned to a commit SHA, and benchmarked in a sandbox so the Lift in percentage points is published alongside a trust score — never behind a paywall, and never influenced by money. Skills, proven. Browse the free directory and join the waitlist.

Why Skill Effects Vary Too Much to Trust Without Measurement

The average skill looks great — until you look at the individual results. That gap is exactly why measurement matters.

According to the SkillsBench study, curated skills raise average pass rates by 16.2 percentage points. But that average hides enormous variance. Domain-level effects range from a modest +4.5pp in Software Engineering to a dramatic +51.9pp in Healthcare — a difference of more than 11x between domains.

More troubling: 16 of 84 tasks showed negative deltas. In nearly one in five benchmark tasks, adding a skill actually made the agent perform worse. Without with-vs-without measurement, a buyer would have no way to distinguish a +51.9pp skill from one that actively hurts performance.

The variance breaks down into three practical takeaways:

• Averages are misleading — a skill's value depends heavily on the domain and task.
• A meaningful share of skills reduce performance, not just fail to help.
• Only controlled comparison (same model, with and without the skill) reveals true Lift.

The research also found that self-generated skills provide no benefit on average — models, as the authors put it, "cannot reliably author the procedural knowledge they benefit from consuming." An AI system cannot reliably grade its own study guide. This is why independent measurement is not just an ethical preference but a methodological requirement.

This finding has direct economic consequences. If skill quality cannot be self-assessed by the systems consuming it, then verification must come from outside — through independent evaluation rather than vendor claims or self-reported ratings. The cost of skipping that step compounds quickly: separate security research found 26.1% of analyzed skills carried at least one vulnerability.

This is the design principle behind SkillzBench's approach: benchmark every skill in a sandbox with the same model, with and without the skill, and publish the Lift in percentage points — always shown separately from ratings, so a skill's measured effect can never be blurred into a marketing score. SkillzBench is building exactly this with-vs-without measurement into its free directory, because the research shows no shortcut can substitute for it.

A skill that helps in one domain can hurt in another, and the only reliable way to know which is which is to measure it directly.

Where the Real Costs of Benchmarking Sit (It's Not Just Compute)

When SkillsBench's authors set out to measure whether agent skills actually help, they started with 322 candidate tasks submitted by 105 contributors — and published just 84. That 74% rejection rate tells you where benchmark money really goes: not into GPUs, but into people and process. The benchmark's methodology paper describes a multi-stage pipeline of automated structural validation, oracle execution where reference solutions must pass 100% of tests, human review on five criteria, and CI-based leakage audits. Each stage is labor that compounds before a single agent run happens.

Human curation is the dominant cost center, and the reasons are methodological, not incidental. The authors note that LLM-generated queries are "confined by the distributions of LLMs, which are the subject of our evaluations" — so tasks must be human-authored, human-reviewed, and human-maintained. Anti-leakage engineering adds another layer: continuous audits to ensure solutions haven't leaked into training data, plus deterministic verification instead of LLM-as-judge, which trades judge-model inference costs for verifier engineering costs.

Then there's the compute multiplier. Because SkillsBench measures augmentation rather than raw capability, every task runs twice — with the skill and without it. Across 7 agent-model configurations, that produced 7,308 valid trajectories. Doubling runs per task is the price of measuring Lift at all.

Token budgets compound across all those trajectories. Anthropic's documentation frames the context window as "a public good" and instructs skill authors to ask "Does this paragraph justify its token cost?" — a concise skill section costs roughly 50 tokens where a verbose one costs 150. The engineering rationale is blunt: "sorting a list via token generation is far more expensive than simply running a sorting algorithm." Benchmark design choices about what agents do in-code versus in-token directly change run cost.

Finally, security verification is now a mandatory overhead. Academic research found 26.1% of 31,132 analyzed skills carried at least one vulnerability, and the ClawHavoc campaign documented $2.3M stolen and $890K in cleanup costs. The hidden cost structure of benchmarking, inferred from methodology:

• Human curation and multi-stage review labor (105 contributors, 322 tasks culled to 84)
• Anti-leakage engineering and deterministic verifier construction
• Security scanning and verification overhead
• Token budgets that compound across 7,308 trajectories
• Doubled compute from with/without-skill runs

This is exactly the cost structure SkillzBench is building its own benchmarking pipeline around — pinning every skill to a commit SHA, security-scanning it, and running it in a sandbox with the same model, with and without the skill, to publish Lift in percentage points. Benchmarking isn't cheap. But as the ClawHavoc figures show, not benchmarking is far more expensive.

Why Benchmarking Is Cheaper Than Incident Response

The cheapest agent skill is the one that fails in a sandbox instead of in production. Every dollar spent on verification and benchmarking is a dollar not spent on incident response, breach remediation, or rolling back a poisoned dependency.

The economics of not benchmarking are now measurable. Independent academic analysis found that 26.1% of 31,132 analyzed skills carried at least one vulnerability, spanning prompt injection, data exfiltration, privilege escalation, and supply-chain risks. Snyk's parallel ToxicSkills research put the figure even higher at 36.82% flawed across 3,984 skills, with 13.4% rated critical. These are not theoretical risks. The ClawHavoc campaign infected 1,184 skills across 12 publisher accounts, reached 247,693 installations, and cost victims $2.3M in stolen cryptocurrency plus $890K in cleanup.

Verification absorbs that overhead once; incident response pays it over and over. A benchmarked, scanned, and license-checked skill either passes or fails before it touches a workflow. The cost of running that pipeline is fixed and predictable. The cost of a breach is not.

The case for measuring Lift, not just trust, is equally concrete. SkillsBench found curated skills raise average pass rates by 16.2 percentage points, but the range runs from +4.5pp in Software Engineering to +51.9pp in Healthcare, and 16 of 84 tasks showed negative deltas. Without a same-model, with-and-without-skill comparison in a sandbox, a buyer cannot tell a +51.9pp skill from one that quietly degrades performance.

That comparison must be independent to be credible. The same research found self-generated skills provide no benefit on average, because models cannot reliably author the procedural knowledge they benefit from consuming. Independence is therefore methodologically required, not merely ethical. When ad money never touches scores and ratings come from reproducible sandbox runs, the numbers survive scrutiny.

SkillzBench is building this layer as a free directory and free MCP connector: every version pinned to a commit SHA, scanned for vulnerabilities, license-checked, quality-judged, and benchmarked in a sandbox with the same model under both conditions. The Lift is published in percentage points, the Trust Score (0-100) is derived from security, quality, license, maintenance, and community signals, and the two are always shown separately. Absorbing verification overhead at the platform level is what keeps that overhead off every individual user's incident budget.

How to Put Measured Skills to Work for Free

The economics of benchmarking might sound like an academic concern — until you realize the alternative is guessing which skills help and which quietly drain your budget or leak your data. The good news is that a verify-first workflow doesn't require you to build any infrastructure yourself. You can borrow the discipline of the benchmark labs and apply it in an afternoon.

Start with the numbers, not the marketing. Research shows skill quality varies wildly: SkillsBench found curated skills raise pass rates by an average of 16.2 percentage points, but the range runs from +4.5pp in software engineering to +51.9pp in healthcare — and 16 of 84 tasks actually got worse with a skill attached. That variance is exactly why a with-skill vs. without-skill measurement, reported as a plain Lift figure, matters more than star ratings or download counts.

Check trust before you check features. The security case is not hypothetical. Independent academic research found 26.1% of 31,132 analyzed skills carried at least one vulnerability, and the OWASP-documented ClawHavoc campaign caused $2.3M in stolen crypto plus $890K in cleanup costs. A composite Trust Score built from security, quality, license, maintenance and community signals is your first filter — before any skill touches a production agent.

A practical verify-first workflow looks like this:

• Browse a curated, deduplicated directory instead of raw GitHub search, so you're comparing vetted candidates.
• Filter by Trust Score first, then compare Lift in percentage points — the measured performance delta, kept separate from ratings.
• Confirm each skill version is pinned to a commit SHA, with license and security checks already done.
• Connect your chosen skills to any agent through a free MCP connector, toggling skills per agent so each one only sees what you've selected.

This is precisely what SkillzBench is building: a free Discover-Verify-Benchmark-Equip-Stay-current pipeline, with a free MCP server to wire verified skills into Claude Code, Cursor, Codex, Gemini CLI or chat-app agents. The platform is pre-launch with a live waitlist, and everything — directory, benchmarks, connector, alerts — will be free, funded by blog advertising that never touches scores.

The benchmark research also offers a design tip worth keeping: focused skills beat comprehensive ones. SkillsBench found tight 2–3 module skills outperform sprawling documentation, echoing guidance that "overly comprehensive skills can hurt more than they help." Pick small, measured, proven skills — and let someone else run the expensive benchmarks.

Skills, proven. Browse the free directory.