Publishing Reproducible Agent Benchmarks

The Benchmark Wild West: Why Nobody Knows Which Skills Actually Help

If you're building agents with Claude Code, Codex, Cursor, or Gemini CLI, you've probably grabbed a handful of SKILL.md instruction packages from GitHub — and you probably have no idea whether any of them actually made your agent better. You're not alone. The skills ecosystem has exploded faster than anyone's ability to evaluate it.

The numbers are sobering. A security analysis of 31,132 agent skills found that 26.1% contain at least one vulnerability, with data exfiltration and privilege escalation among the most common patterns. Snyk's separate ToxicSkills research put the figure even higher: 36.82% of 3,984 skills contained security flaws.

And even when a skill is safe, "does it help?" turns out to be a surprisingly hard question. The first benchmark to treat skills as first-class evaluation artifacts found that curated skills lift pass rates anywhere from +4.5 percentage points in software engineering to +51.9 points in healthcare — but on 16 of 84 tasks, curated skills actually made agents worse. The same skill can be a superpower in one domain and a liability in another.

So why can't you just look this up for the skill you're considering? Three reasons:

• Most leaderboards measure accuracy alone, ignoring cost — AstaBench found one agent setup cost 4x more per problem than another while scoring half as much.
• Single-run results overstate reliability: GPT-4o dropped from under 50% success to roughly 25% when tasks had to succeed eight consecutive times.
• Each skill author, marketplace, and blog post evaluates in its own bespoke way, so numbers are never comparable across sources.

Accuracy-only, apples-to-oranges evaluation can't answer the question you actually have: does this skill help my agent, on my tasks, at an acceptable cost? Answering it requires holding everything constant — the same model, the same sandbox, tested with and without the skill — and publishing the difference, the Lift, in plain percentage points.

That's the gap SkillzBench is being built to close. It's an independent testing lab that will benchmark every skill under identical conditions, publish Lift separately from any quality or trust ratings, and verify skills before they're ever scored — because with a quarter of skills carrying vulnerabilities, evaluation without verification would just be ranking broken things.

Until skills are tested the way medicines are — same trial, same conditions, effects reported honestly, including when the effect is negative — the skills ecosystem stays what it is today: a wild west where the loudest README wins.

The Reproducibility Playbook: What Rigorous Benchmarks Do Differently

The difference between a benchmark you can trust and a leaderboard you scroll past comes down to design choices made before a single agent ever runs. When you line up AstaBench, SkillsBench, SWE-bench, REPRO-Bench, and 𝜏-bench, a shared playbook emerges — and it treats reproducibility as a first-class requirement, not a footnote.

The first pillar is the containerized environment. SWE-bench introduced Dockerization in August 2024 explicitly for easier evaluation, and SkillsBench runs every task in a standardized container. If the environment drifts between runs, the numbers drift with it, and no amount of post-hoc analysis fixes that.

The second pillar is deterministic verification. SkillsBench avoids LLM-as-judge entirely, requires oracle solutions to pass 100% of tests, and logs full trajectories. 𝜏-bench takes a similar route, evaluating database state after task completion rather than judging conversations. Objective scoring is what makes a result worth publishing twice.

The third pillar is confound control. Ai2's framework for AstaBench is explicit about this: report computational cost and tool usage, standardize task interfaces, and control knowledge with date-restricted corpora. Their reasoning is blunt — "current leaderboards fail to properly account for these variables." The cost dimension matters more than most teams expect. AstaBench found a ReAct agent powered by gemini-flash cost 4x more per problem than o3 while scoring half as much, which is why accuracy-only rankings mislead.

The fourth pillar is honest reporting, including negative results. REPRO-Bench published that its best baseline agent scored 21.4% — below random guessing. SWE-bench tags which leaderboard runs were verified by its own team. The benchmarks that earn trust are the ones willing to show what failed.

The fifth pillar is leakage prevention. SkillsBench's authors insist that skills must be applied to a class of tasks, not a single instance, and provide procedural guidance rather than declarative answers. A benchmark that a skill can memorize its way through measures nothing.

The playbook, condensed:

• Containerize everything, so the environment is identical across runs
• Verify deterministically — state checks and test suites, not model opinions
• Control the model, cost, and tool access, and pin versions so results stay comparable
• Open-source the code and data, as REPRO-Bench and 𝜏-bench both did
• Publish negative results with the same prominence as positive ones

The stakes are real. SkillsBench found curated skills lift pass rates by an average of +16.2 percentage points, but 16 of 84 tasks showed negative deltas — the same skill can help in one domain and hurt in another. That variability is exactly why SkillzBench is building its benchmark methodology around this playbook: same model, same sandbox, with and without the skill, Lift reported per run. When a number can move a decision, the burden is on the benchmark to prove it means something.

Lift, Not Vibes: Measuring What a Skill Actually Changes

A skill that lifts performance in one domain can quietly degrade it in another. That is not a hypothetical — it is the headline finding of the first benchmark to treat agent skills as first-class evaluation artifacts, and it is why "vibes" are not enough when deciding which skills to attach to an agent.

The methodology that reveals this is simple in principle: run the same tasks on the same model twice, once with the skill and once without, and publish the difference — the Lift — in percentage points. When researchers applied this rigorously, they found curated skills lift pass rates by anywhere from +4.5 percentage points in software engineering to +51.9 percentage points in healthcare, averaging +16.2pp. The spread matters more than the average.

Three lessons from that work shape how SkillzBench is building its Benchmark pillar:

• Show Lift per domain, never as a single number. In the same study, 16 of 84 tasks showed negative deltas from curated skills — an average of +16.2pp would mask skills that help in one domain and hurt in another.
• Report cost alongside accuracy. The AstaBench evaluation found a ReAct agent powered by gemini-flash cost 4x more per problem than o3 while scoring half as much — accuracy-only leaderboards mislead.
• Pin the exact model version for every run. Lift measured on one model snapshot is not comparable to results on another, especially as provider prices and models change.

The cost lesson deserves emphasis. Ai2's team built their benchmark around a Pareto frontier of accuracy versus cost, arguing that "current leaderboards fail to properly account for these variables" — meaning tool usage and computational cost. A skill that adds +10pp but doubles token spend per task may be a bad deal; a leaderboard that hides cost cannot tell you.

Deterministic scoring is the other half of reproducibility. SkillsBench's authors used containerized environments, deterministic verifiers rather than LLM-as-judge, and full trajectory logging with leakage audits — and required oracle solutions to pass 100% of tests. Sierra's τ-bench research reached a similar conclusion, evaluating database state after task completion to avoid any human or LLM-based judging. Objective scoring is what makes a benchmark result something you can trust and re-verify later.

This is also why averages mislead in both directions. A skill with a modest average Lift might be exceptional in healthcare and worthless in code — or actively harmful in 16 of 84 tasks. SkillzBench's approach is to test every skill in a sandbox with the same pinned model, with and without the skill, and publish domain-level Lift separately from trust ratings, so efficacy evidence and safety signals never blur together.

If you want skill recommendations grounded in measured Lift rather than vibes, join the SkillzBench waitlist — we publish what each skill actually changes, per domain, with cost in view.

Social proof text: The skills security research that found ~26% of skills carry at least one vulnerability is exactly why SkillzBench measures Lift with pinned models, per domain, before you attach anything to your agent.

Verify First, Benchmark Second: Security Gates Before Performance Claims

A skill that leaks your data can't be a skill worth benchmarking. That's the uncomfortable math behind recent agent-skill research: before any performance number can mean anything, the artifact itself has to be safe to run.

The security case for sequencing is stark. One large-scale analysis found that 26.1% of 31,132 analyzed skills contain at least one vulnerability, with data exfiltration (13.3%) and privilege escalation (11.8%) the most common patterns. Script-bearing skills fare far worse — they're 2.12x more likely to carry vulnerabilities than instruction-only packages.

The threat isn't hypothetical. OWASP's case studies document the ClawHavoc campaign, in which attackers published 1,184 malicious skills through 12 compromised publisher accounts, racking up 247,693 confirmed installations and $2.3M in stolen cryptocurrency. At the peak, five of the seven most-downloaded skills on the marketplace were malicious.

ClawHub's post-incident response reads like a checklist for what should have existed beforehand: mandatory security reviews, publisher identity verification, download velocity monitoring, and automated malware scanning. OWASP's prevention framework now recommends pre-publication security review, automated vulnerability scanning, and permission-based execution as baseline controls.

This is why SkillzBench sequences its Verify pillar before Benchmark. Every skill gets pinned to a specific commit SHA so results are attributable to an exact version, then passes through security scanning, license checks, and quality validation before it ever reaches a sandbox. And the launch scope reflects the risk data directly: instruction-only skills (SKILL.md plus text references) are benchmarked, while script-bearing skills — the category with the elevated vulnerability rate — are listed link-out only.

The gates serve the numbers, not just safety. A Lift figure published from a compromised skill is worse than no figure at all, because it lends credibility to an artifact that may be exfiltrating data mid-run. The security researchers behind the vulnerability study put it plainly: there's an "urgent need" for mandatory security vetting before this attack vector is exploited further.

Verification also protects reproducibility. Pinning versions to commit SHAs means a benchmark result can be tied to precisely what was tested — and when upstream skills change, re-scanning and re-benchmarking keeps published numbers honest rather than stale.

Trust in a benchmark starts with trusting what goes into it. SkillzBench is building exactly that pipeline, and you can join the waitlist to follow its progress.

ctaText: Join the SkillzBench waitlist socialProofText: Grounded in published research on skill security and benchmark methodology — not vendor claims.

Building It: Versioned Runs, Transparency Tiers, and Staying Current

A benchmark that goes stale the moment a skill's author pushes a commit isn't a benchmark — it's a screenshot. The hard part of publishing agent evaluations isn't running them once; it's keeping them honest as models, skills, and prices shift underneath you.

The strongest precedent for this comes from benchmarks that version everything. SWE-bench tags which leaderboard runs were "performed or directly checked by the SWE-bench team," so readers can distinguish verified results from self-reported ones. Ai2's AstaBench goes further, using transparency tiering to flag non-comparable submissions — configurations with privileged tool access or unusual cost profiles sit apart from the controlled tier. That separation matters because accuracy-only leaderboards mislead: AstaBench found a ReAct agent powered by gemini-flash cost four times more per problem than o3 while scoring half as much.

For skill benchmarks, the stakes are higher because skills themselves change upstream. A dated, version-pinned re-run — same model version, same sandbox, same task suite, with the skill pinned to a specific commit — is the only way to know whether a Lift number still holds. SkillsBench's findings make this vivid: curated skills lifted pass rates anywhere from +4.5pp in Software Engineering to +51.9pp in Healthcare, but 16 of 84 tasks showed negative deltas. A skill that helped last month can quietly hurt after an upstream edit, and only re-benchmarking catches it.

A practical publication standard for staying current looks like this:

• Pin every re-benchmark run to a date, model version, and skill commit SHA, and publish deltas against the prior run rather than silently replacing old numbers.
• Tag who ran each evaluation — lab-verified, team-checked, or self-reported — so comparability is visible at a glance.
• Apply transparency tiering so non-comparable configurations (different tools, different costs) never contaminate the main comparison.
• Log full trajectories for every run, following SkillsBench's practice of deterministic verifiers and leakage audits, so disputed results can be inspected rather than argued about.

Reliability data reinforces why repeated, logged runs beat one-shot scores. Sierra's 𝜏-bench work showed GPT-4o dropping from under 50% average success to roughly 25% on pass^8 in 𝜏-retail — a 60% fall from pass^1. Single-run results systematically overstate what users will experience.

This is exactly the discipline SkillzBench is building into its Stay-current pillar: tracking upstream skill changes, re-scanning for security issues, re-benchmarking against the pinned baseline, and notifying users when a skill's Lift moves. Given that published research found 26.1% of analyzed skills carry at least one vulnerability, "current" has to mean re-verified, not just re-crawled. A Lift number is a claim about a specific version at a specific moment — and the platform's job is to make that moment visible, repeatable, and revisable.