How Are Agent Skills Benchmarked?

Why Agent Skills Need Their Own Benchmarks (Not Just Model Tests)

Your model just aced the benchmark — and that tells you almost nothing about whether the skill you installed made your agent better or worse. That gap is the core problem facing anyone deploying agent skills today.

Existing benchmarks answer one question: "How well can this model perform task X?" They cannot answer the question that actually matters when you install a skill: "How much does skill Y improve performance on task X?" That motivating gap is why researchers behind the SkillsBench paper built the first benchmark treating skills as first-class evaluation artifacts rather than invisible accessories.

The clearest way to think about this comes from a three-layer mental model gaining traction across the field:

• Foundation models are CPUs — raw compute, measured by model benchmarks like SWE-bench.
• Agent harnesses are operating systems — Claude Code, Codex, Cursor, Gemini CLI orchestrate the model's capabilities.
• Skills are applications — and nobody judges an operating system by assuming every app helps.

As Anthropic's engineering team puts it, "when we evaluate 'an agent,' we're evaluating the harness and the model working together" — and skills add yet another layer that traditional evals never isolate (Anthropic).

Here's the unsettling part: skills can hurt. In the SkillsBench results, 16 of 84 tasks showed negative deltas — installing the skill made the agent perform worse than running bare. Curated skills delivered a +16.2 percentage-point average improvement, but the benefit ranged wildly by domain, from +4.5pp in software engineering to +51.9pp in healthcare. Averages hide landmines.

The risk isn't only performance. Security scanning has become standard practice precisely because skills are instruction packages you're injecting into an agent's context — and published research found roughly 26% of skills carry at least one vulnerability. NVIDIA's SkillEvaluator runs every skill through prompt-injection, data-exfiltration, and PII detection before it earns a score (NVIDIA).

The stakes differ by role. Business owners need to know a skill won't silently degrade an automated workflow — or leak data. Developers need to catch regressions before shipping. Prompt engineers need to understand that a single wording change can flip a skill from helpful to harmful, as Langfuse discovered when one comment adjustment eliminated a failure pattern present in over 90% of test cases (Langfuse).

This is why SkillzBench is building skill-level benchmarking — sandboxed with/without runs, published Lift, and security scanning — rather than relying on model scores. The model tells you what's possible; only skill benchmarks tell you what was added.

The Skill Lift Methodology: Controlled A/B With vs. Without the Skill

The question "is this skill any good?" sounds subjective — until you run the same agent on the same tasks twice, once with the skill and once without. That controlled A/B design has become the de facto standard for measuring agent skills, used by everyone from NVIDIA to academic benchmark teams.

The setup is deliberately simple. The agent runs in an isolated sandbox with an identical prompt, model, inputs, and grading criteria — the only variable is whether the skill is installed, as NVIDIA's SkillEvaluator methodology describes. Because everything else is held constant, any score difference can be attributed to the skill itself.

That difference is called Skill Lift: the with-skill score minus the without-skill score, expressed in percentage points. It answers the question traditional model benchmarks ignore — not "how well can this model do task X?" but "how much does this skill actually help?" The SkillsBench leaderboard reports the same paired delta across dozens of agent–model configurations.

The numbers show why this matters. NVIDIA reported an average Correctness Lift of +41 points (46→87) across 300+ skills, with Lift varying from +2 to +46 points depending on the product. The academic SkillsBench study found curated skills deliver a +16.2 percentage-point average improvement — but 16 of 84 tasks showed negative deltas, meaning some skills actively make agents worse.

• NVIDIA: +41-point average Correctness Lift, +39 Effectiveness, +35 Efficiency
• SkillsBench: +16.2pp average gain, ranging from +4.5pp (software engineering) to +51.9pp (healthcare)
• Lift spread across products: +2 to +46 points — the skill and task matter far more than the harness
• Roughly 19% of tested tasks showed negative Lift, proving skills can hurt

Lift is the metric that matters because it isolates the skill's contribution. A skill with impressive-sounding documentation but a Lift near zero — or below zero — is decoration at best and a liability at worst. That's why SkillzBench is building its benchmarking pillar around exactly this design: every skill tested in a sandbox with the same model, with and without the skill, with the Lift published in percentage points.

One principle is non-negotiable: Lift must be reported separately from trust ratings. A skill can be safe, well-licensed, and well-maintained yet add nothing — or be effective but carry vulnerabilities. Collapsing those axes into a single number hides exactly the information users need. Keeping them side by side, but never merged, is what makes a benchmark honest.

Skills, proven. — see the Lift for yourself on SkillzBench.

How Skills Are Actually Graded: Deterministic Checks, LLM Judges, and the Triggering Problem

Grading an agent skill isn't as simple as checking whether a task succeeded — it requires answering two different questions at once: did the agent actually do the required actions, and did it do them the right way? The industry has converged on a hybrid answer to both.

The consensus approach layers two grader types. OpenAI's guidance on evaluating skills recommends deterministic checks over JSONL event streams — "did it run npm install?" — because, as they put it, "everything is deterministic and debuggable." But deterministic checks only answer "did it do the basics?" For qualitative questions, OpenAI adds a model-assisted rubric step, and Langfuse's skill evaluation case study agrees: checking whether instrumentation is correct "can't be done by string match," so an LLM judge compares the agent's changes against expected output. Anthropic frames the same tradeoff by categorizing graders as code-based, model-based, or human, each with distinct costs and weaknesses.

Not everyone embraces the judge, though. The SkillsBench academic benchmark deliberately uses fully deterministic verification to avoid LLM-as-a-judge variance — a notable methodological divergence from Ai2's AstaBench, which grades against per-problem rubrics using the LLM-as-a-judge paradigm. There's no single right answer yet, and the split is worth watching.

The axis most teams miss entirely is triggering and discoverability. A skill that never gets invoked has a Lift of zero, no matter how good its instructions are. OpenAI notes that the name and description fields in a skill's YAML front matter are "the primary signals Codex uses to decide whether to invoke the skill at all" — if they're vague or overloaded, the skill won't trigger reliably. Langfuse learned this the hard way: when they made a skill's description more abstract, the skill stopped being invoked entirely. Their takeaway: "Specific keyword matching is super important." NVIDIA takes the axis seriously enough to score Discoverability as one of five dimensions, warning that "a skill that loads when it is not relevant can reduce agent performance."

Practically, that means when evaluating a skill you should check:

• Whether the description uses concrete keywords matching how users actually phrase tasks
• Whether the name and description are specific rather than over-abstracted
• Whether the skill competes badly with other installed skills for the agent's attention
• Whether it ever loads when it shouldn't, dragging down unrelated tasks

Finally, cost and efficiency deserve their own scoreboard. NVIDIA found that "token savings are not automatic" — one skill, cuopt-install, actually increased token usage by 120.3% and execution time by 20.8% — so SkillEvaluator tracks tokens and execution time separately from correctness scores. AstaBench goes further, plotting a Pareto frontier of accuracy versus cost rather than collapsing the two into one number.

This is exactly why SkillzBench is building its benchmarking to report Lift in percentage points from sandboxed with/without runs, with efficiency tracked as its own axis — so you can see, before you install, whether a skill actually helps, hurts, or just burns tokens. Skills, proven.

Security, Trust, and Reproducibility: What to Check Before You Install Any Skill

Installing an agent skill without verification is akin to handing over your keys to an unvetted stranger. Published research indicates that approximately 26% of skills carry at least one vulnerability, making pre-install security checks non-negotiable.

NVIDIA’s SkillEvaluator framework defines a rigorous Tier 1 standard for this verification layer. It mandates strict schema validation and quality scoring before any skill enters the sandbox.

The framework specifically scans for prompt injection attempts and data exfiltration vectors. It also detects Personally Identifiable Information (PII) and leaked secrets that could compromise your infrastructure.

License compliance is another critical checkpoint. Skills must adhere to open-source standards to avoid legal liabilities for your organization. This ensures the code you run does not violate intellectual property rights.

Nous Research’s SkillSpector demonstrates the speed of these checks. It runs 29 passing tests in roughly 1.5 seconds per scan. This rapid feedback loop allows developers to iterate quickly without sacrificing security.

Reproducibility infrastructure separates a genuine benchmark from a casual vibe check. Without it, results are untrustworthy and impossible to verify.

Key components of robust reproducibility include:

• Docker sandboxes that isolate execution environments completely
• Commit-pinned data snapshots to ensure identical inputs
• Deterministic verifiers that eliminate grading variance
• Leakage audits that detect data contamination in CI

NVIDIA pins its evaluation data to a specific commit hash, ensuring every run starts from the same baseline. This level of control is essential for trusting the reported metrics.

For readers who do not wish to build this stack themselves, SkillzBench is developing a practical shortcut. Its Verify pillar automates these complex checks into a single workflow.

The platform generates a 0–100 Trust Score based on security, quality, and license signals. This metric provides a clear, at-a-glance indicator of skill reliability.

SkillzBench ensures that money never touches these scores, maintaining independence. This integrity is crucial for a free, independent directory of AI agent skills.

By combining rigorous security scanning with transparent trust scoring, the platform lowers the barrier to safe adoption. Users can confidently equip their agents with proven capabilities.

Skills, proven. Browse and test free agent skills on SkillzBench.

How to Benchmark Your Own Skills (and What to Do With the Results)

You don't need a lab, a budget, or a hundred test cases to find out whether a skill actually helps your agent. You need ten to twenty prompts, one fixed model, and a habit of running the same tasks twice — once with the skill installed, once without.

That small-benchmark approach is exactly what OpenAI recommends: for a single skill, a small set of 10–20 prompts is enough to surface regressions and confirm improvements early. The rigor comes from the design, not the size. Every serious test should cover three conditions: no skill, curated skill, and self-generated skill. That third condition matters because the SkillsBench paper found self-generated skills provide no benefit on average — models cannot reliably author the procedural knowledge they benefit from consuming.

Iteration is where the payoff lives. Langfuse documented a failure pattern appearing in over 90% of test cases that vanished the moment one comment changed from "optional" to "mandatory." Their loop is simple and repeatable:

• Run the benchmark and record the baseline score.
• Read individual traces, not just the numbers — the score tells you something broke, the trace tells you why.
• Change one thing: a comment, a constraint, a description.
• Re-run and compare the delta. If it moved, keep it; if not, revert.
• Watch for the skill no longer triggering at all — over-abstracting a description can do exactly that.

Then act on the numbers honestly. Report Skill Lift separately from quality and security ratings, because they measure different things — NVIDIA's evaluation found token savings are not automatic, with one skill increasing token usage by 120.3%. Watch for negative deltas: 16 of 84 tasks in SkillsBench showed skills actively hurting performance, so a skill that scores well on paper can still drag your agent down in practice.

You don't have to build the infrastructure yourself, either. SkillzBench is building a free directory and MCP connector that pins each skill to a commit SHA, scans it, benchmarks it in a sandbox, and publishes the Lift — so the small-benchmark discipline above becomes a starting point rather than a chore. Skills, proven.