Why Every Skill Review Needs a Lift Metric

The Problem: You Can't Judge an Agent Skill by Looking at It

Agent skills went from a niche Claude Code feature to an industry-wide paradigm in roughly six months — and in that entire time, nobody has built a reliable way to tell whether a skill actually helps. As the SkillsBench research team puts it, despite rapid adoption, there is no standard way to measure whether skills actually help, leaving practitioners unable to make informed adoption decisions.

The problem is that every review mechanism we rely on today — star ratings, READMEs, GitHub stars, author reputation — is a form of inspection. And inspection turns out to be a terrible predictor of performance.

Consider what happens when you actually measure. In controlled benchmark testing, curated skills raise average pass rates by 16.2 percentage points — a genuinely strong result. But 16 of 84 tasks (19%) showed negative deltas, meaning the skill made the agent measurably worse. On average a skill helps; on nearly one in five tasks, it actively harms.

It gets worse. An independent practitioner study found that a skill built directly from correct, human-written documentation caused the model to produce incorrect output 66% of the time — on a task the same model solved correctly with no skill at all. The source material was flawless. The skill still broke the model. And the failure modes turned out to be "more surprising and less predictable" than expected, with six distinct interference mechanisms identified across a review of 673 ecosystem skills.

Then there is the safety dimension. A large-scale security analysis of 31,132 skills found that 26.1% contain at least one vulnerability, with 5.2% showing high-severity patterns suggesting malicious intent. Skills execute with implicit trust and minimal vetting — and script-bundling skills carry 2.12× the vulnerability odds of instruction-only ones.

So inspection-based review fails on both fronts:

• Value is invisible. A well-documented, popular skill can still degrade output on 19% of tasks, and nobody can tell which tasks without running them.
• Risk is invisible. Over a quarter of skills carry vulnerabilities that READMEs and star ratings never surface.
• Failure is unpredictable. Even correct documentation can produce a skill that breaks a task the model handled fine alone.

This is the gap SkillzBench is being built to close: rather than asking whether a skill looks good, its benchmarking pipeline will run every skill in a sandbox with the same model, with and without the skill installed, and publish the resulting Lift in percentage points. The only variable is the skill itself — which is exactly how NVIDIA's SkillEvaluator and the Tessl evaluation framework define the measurement too.

The conclusion is unavoidable: you cannot judge an agent skill by looking at it. You have to measure the delta.

The Solution: Measure Lift, Not Vibes

If four independent teams — academic researchers, a GPU vendor, an evaluation framework, and a solo practitioner — all land on the same answer, it is probably the right one. When it comes to measuring agent skills, they do: run the same task with and without the skill in a controlled sandbox, and report the delta.

NVIDIA formally defines this delta as Skill Lift — "the difference between the with-skill and without-skill scores," measured in isolated sandboxes where skill installation is the only variable (NVIDIA's SkillEvaluator documentation). The academic SkillsBench benchmark takes the same approach across three conditions: no skills, curated skills, and self-generated skills (the SkillsBench paper). Tessl's framework solves each task twice to estimate "the marginal value of the skill" (its SIGKDD workshop paper), and Dachary Carey's independent recommendation is blunt: "you need to really test the no skill vs. with skill lift, with realistic context, across a variety of conditions and tasks" (her practitioner analysis).

The numbers this methodology produces are far more honest than any star rating. Curated skills raise average pass rates by 16.2 percentage points — but that average hides enormous variation:

• Domain lift ranges from +4.5pp in Software Engineering to +51.9pp in Healthcare (SkillsBench)
• Per-product lift in NVIDIA's testing ranged from +2 to +46 points, while harness differences averaged only ~5 points — "the domain, the task, and the evaluation design mattered more than the harness"
• 16 of 84 tasks (19%) showed negative deltas: the skill actively hurt performance
• One documentation-derived skill caused 66% incorrect output on a task the model completed correctly without it

That last two points carry a critical design implication: a secure skill can still have negative lift. Security scanning and performance measurement answer different questions, and collapsing them into one score erases the failure mode users most need to see. A skill can pass every vulnerability check and still degrade output; another can boost accuracy while quietly increasing token consumption — NVIDIA found one skill raised tokens by 120.3% while another cut them by 76.9%.

This is exactly why SkillzBench is building its Benchmark pillar around with/without-skill sandbox testing, publishing Lift in percentage points — and why Lift stays visually separate from the 0–100 trust score. Trust tells you whether a skill is safe to install. Lift tells you whether it's worth installing. Only together do they answer the question every skill review should start with: does this skill actually help?

What a Real Lift Review Must Include

A single lift number can flatter a skill in exactly the ways that matter least. Because lift swings so wildly by domain — from +4.5pp in Software Engineering to +51.9pp in Healthcare, per SkillsBench findings — a headline average tells you almost nothing about whether a skill helps with your tasks. A real lift review therefore publishes per-domain and per-task breakdowns, so you can judge relevance to your actual workload. NVIDIA's evaluation of 300+ skills saw per-product lift range from +2 to +46 points, while the harness (Claude Code vs. Codex) accounted for only ~5 points of difference — the task and domain dominate, not the tooling (NVIDIA SkillEvaluator data).

Second, lift must include the cost dimension: token and time deltas. A skill can win on accuracy while quietly doubling your spend — NVIDIA measured one skill (cuopt-install) raising token consumption 120.3% and execution time 20.8%, while another (jetson-optimize-memory) cut tokens 76.9%. Accuracy lift without efficiency lift is only half the review.

Third, negative-lift findings deserve prominent flagging, not burial in an appendix. Curated skills showed negative deltas on 16 of 84 tasks (19%), and one documentation-derived skill caused a model to produce incorrect output 66% of the time when unassisted it succeeded (independent practitioner testing). These failures are, as that author put it, "difficult to detect for an average user."

A rigorous lift review should therefore include:

• Per-domain and per-task lift breakdowns — never one headline number
• Token and execution-time deltas, published alongside accuracy lift
• Explicit negative-lift flags wherever the with-skill run underperformed the baseline
• Extra caution for script-bearing skills, which carry 2.12× the vulnerability odds of instruction-only skills (a security study of 31,132 skills)

That last point shapes how SkillzBench is building its own reviews: launch scope covers instruction-only skills, with script-bearing skills listed link-out only pending sandboxed benchmarking.

Finally, lift reviews should reward focus and punish bloat. SkillsBench found that focused skills with 2–3 modules outperform comprehensive documentation-style skills, and that self-generated skills — ones the model wrote for itself — show no average benefit at all. A skill that reads like complete documentation is often a worse bet than a tight, targeted one, and "comprehensive" is not a proxy for effective. Measure the delta, per task, with costs included — everything else is guesswork.

Lift Is Not Static: Why Re-Benchmarking Matters

The lift number you published last month might already be wrong. Tessl's evaluation of 500 skills across 19 agent-model configurations found that models vary widely in how closely they follow skill instructions, producing substantial differences in performance gains — meaning the same skill can deliver very different lift depending on what's running underneath it.

This is a structural problem, not an edge case. Models get updated constantly, agent platforms evolve, and the Agent Skills spec itself remains unversioned. Dachary Carey, whose independent analysis documented a documentation-derived skill causing 66% incorrect output in a model that succeeded unassisted, treats skills as "a full product surface" requiring continuous re-evaluation because products, models, and platforms all change. A skill that helped in March can quietly degrade by September.

What makes this especially dangerous is that failure is unpredictable. Her review of 673 ecosystem skills found six distinct content interference mechanisms, and the failure modes were "more surprising and less predictable" than expected. Most users will never notice the regression — they'll just get worse results and blame the model.

So a credible skill review can't be a snapshot. It needs an ongoing pipeline:

• Track upstream changes — a skill's source repo is live, and edits can change behavior without any version bump.
• Re-scan for security and quality issues whenever the skill or its dependencies shift.
• Re-benchmark lift against current models, since the baseline agent itself improves over time.
• Notify users when a previously positive lift turns negative or marginal.

This is exactly the problem SkillzBench's Stay-current pillar is being built to address: continuous upstream tracking, re-scanning, re-benchmarking, and notification, so a skill's published lift reflects the world as it is, not as it was when first tested. It pairs with the Benchmark pillar's methodology — the same model, in a sandbox, with and without the skill — so re-runs stay comparable over time.

The alternative is flying blind between model releases. Without re-benchmarking, every lift score carries an invisible expiration date, and nobody reading a review can tell whether it's still valid. In an ecosystem where roughly one in four skills carries at least one vulnerability and lift can swing from +46 points to negative depending on conditions, static reviews aren't just incomplete — they're misleading.

CTA text: Skills, proven. Join the SkillzBench waitlist to get lift-tested skill reviews the moment we launch. Social proof text: Grounded in independent research from SkillsBench, NVIDIA SkillEvaluator, and practitioner evals — not vendor marketing.

How SkillzBench Puts Lift at the Center of Every Review

The research verdict is clear: skills must be measured as a delta, not judged by reputation. SkillzBench's Benchmark pillar is being built to match that emerging standard exactly — because four independent methodologies, from an academic benchmark to NVIDIA's SkillEvaluator, have already converged on the same answer.

Every skill SkillzBench tests will run in a sandbox with the same model, with and without the skill installed, and the result will be published as Lift in percentage points. That's precisely how SkillsBench measured its +16.2pp average lift across curated skills, and how NVIDIA formally defines Skill Lift as the difference between with-skill and without-skill scores. The design principle behind it is simple: isolate the skill as the only variable.

Just as important is what stays separate. Lift is always displayed apart from the 0–100 trust score, and money never touches scores. A skill can be secure and still hurt performance — SkillsBench found 19% of tasks showed negative deltas, and one documentation-derived skill caused incorrect output 66% of the time when the model succeeded unassisted. Conversely, a high-lift skill could carry risk: a security study found 26.1% of analyzed skills contain at least one vulnerability. Conflating those two dimensions would mislead everyone.

The full verification stack around each benchmark includes:

• Version pinning to a commit SHA, so every Lift number refers to an exact artifact
• Security scanning and license checking before anything gets benchmarked
• Re-benchmarking on upstream changes, since lift isn't static — models and skills both drift
• Link-out-only listings for script-bearing skills, which carry 2.12× higher vulnerability odds than instruction-only skills

That last point reflects the launch scope: instruction-only skills (SKILL.md plus text references) get full benchmarking, while script-bearing skills are catalogued but not executed. And because NVIDIA's data shows token costs can rise 120.3% with a skill installed, efficiency deltas belong in the report alongside accuracy.

SkillzBench is pre-launch and actively being built. Join the waitlist to see the first Lift reports when they publish — skills, proven.