Are Claude Skills Worth It? What the Benchmarks Actually Show

The Evidence Gap: Everyone Uses Skills, Nobody Measures Them

Here's the uncomfortable truth about Claude Skills: nobody outside Anthropic has published independent numbers showing they actually improve task outcomes. We dug through Anthropic's engineering blog, its platform documentation, and the academic literature, and the performance side of the "worth it" equation is essentially blank. The security side, meanwhile, is thoroughly quantified.

That asymmetry should stop you. The first large-scale empirical study of agent skills analyzed 31,132 unique skills and found 26.1% contain at least one vulnerability, spanning prompt injection, data exfiltration, privilege escalation, and supply chain risks. So we know what skills cost. What we don't know, publicly, is what they deliver.

To be fair, Anthropic has invested heavily in testing infrastructure. Skill Creator 2.0 ships with a Grader, a Blind Comparator, an Analyzer, and a workflow that explicitly compares "skill" versus "no skill" baselines, as Daniel Nest's hands-on review documents. That's the right methodology. The problem is what happens inside it.

Nest's critique cuts deep: Anthropic provides only "two short paragraphs of vague instructions" for writing assertions, leaving eval metrics "up to Claude's intuition." His conclusion is worth pinning above your desk: every "your skill improved by 40%" stat is only as good as the quality of these assertions. That 40% figure, notably, appears only as a hypothetical in his piece. No real numbers were disclosed.

Anthropic itself adds caveats that make attribution even harder. Its engineering guidance notes that agent benchmarks measure the harness and model together, and that static evals can even mislabel superior behavior as failure. Its best-practices docs state plainly that "effectiveness depends on the underlying model" and that "what works perfectly for Opus might need more detail for Haiku."

If you're deploying skills today, you're likely doing so without evidence on three fronts:

• Performance: no public, independent skill-vs-baseline Lift numbers exist in any source we reviewed.
• Security: 26.1% of analyzed third-party skills carry at least one vulnerability, and script-bundling skills are 2.12× riskier than instruction-only ones.
• Token cost: Anthropic calls the context window "a public good" and asks authors to justify every paragraph's token spend, yet nobody publishes that math per skill.

The gap between adoption and measurement is where risk lives. This article is an honest audit of what the evidence supports and what it doesn't. Where published data runs out, we'll say so, and we'll point you to places doing the measurement work in the open, like SkillzBench's free sandboxed benchmarks, which run the same model with and without each skill and publish the Lift. Faith is fine for religion. For software you attach to your agent, you deserve numbers.

Why Skill Benchmarks Are Harder Than They Look

You likely assume a benchmark tells you if a skill works. In reality, it often tells you nothing at all. The measurement problem is more complex than most developers realize.

Anthropic explicitly notes that agent benchmarks evaluate the harness and model together. This means you cannot isolate the skill’s specific contribution from the underlying infrastructure. The system acts as a single unit during testing.

Consider the Opus 4.5 flight-booking loophole. The model found a policy gap to solve a task. It "failed" the static evaluation because it did what the test expected. Yet it actually delivered a better solution for the user. Static evals frequently misgrade genuinely superior behavior as failure.

This creates a fundamental attribution error. If the benchmark measures the whole system, how do you know the skill helped? You don’t. The result reflects the combined output of the code, the prompt, and the model’s intelligence.

Model dependence adds another layer of confusion. Anthropic’s documentation states that skill effectiveness depends on the underlying model. A skill that performs perfectly on Opus might fail on Haiku. What works for one model may need more detail for another.

A result on a "hero" model does not transfer to your specific setup. You must test the skill on the exact model you use in production. General claims about performance are often misleading without this context.

So, what does a trustworthy benchmark look like? It requires a controlled environment. The methodology must be rigorous to isolate variables.

Here is what valid skill benchmarking requires:

• Same model used for both test runs
• Identical task prompts and conditions
• One run with the skill active
• One run without the skill (baseline)
• Clear, objective success criteria for grading

This A/B approach reveals the actual Lift. It shows whether the skill adds value or just consumes context tokens. Anthropic advises that the context window is a public good. Every token must justify its cost.

Independent platforms like SkillzBench now run these specific tests. They benchmark skills in a sandbox with and without the skill. This provides the data you need to make informed decisions.

You can view these verified Lift results for free at skillzbench.com. The platform ensures you see real performance data, not just marketing claims. This transparency is essential for safe adoption.

The Cost Side Is Measured: Security and Token Economics

When weighing whether Claude Skills are worth adopting, the security and token-economics side of the equation is the one area where the evidence is concrete and large-scale. The performance side remains largely unmeasured in public sources, but the cost side has been quantified in ways that should change how teams approach skill selection.

A first-of-its-kind academic study analyzed 31,132 unique skills collected from two major marketplaces and found that 26.1% contain at least one vulnerability. The vulnerabilities span 14 distinct patterns across four categories: prompt injection, data exfiltration, privilege escalation, and supply chain risks. This isn't theoretical risk. The same study documented a benign-appearing "GIF Creator" skill that silently downloaded and executed MedusaLocker ransomware.

The risk profile sharpens considerably when you look at skill structure. Skills that bundle executable scripts are 2.12 times more likely to contain vulnerabilities than instruction-only skills (OR=2.12, p<0.001), according to the same research. And 5.2% of analyzed skills exhibited high-severity patterns that strongly suggest malicious intent rather than accidental flaws.

For comparison, the study found 7.2% of MCP servers carry vulnerabilities, with 5.5% showing tool poisoning patterns. Skills sit in a similar risk band, but the attack surface is different: skills execute code with high trust and minimal scrutiny, as the researchers put it.

The security picture matters because it sits alongside a token-economics reality that Anthropic itself flags. Official best-practices documentation describes the context window as "a public good," meaning a skill's tokens compete with system prompts, conversation history, and other skills' metadata. Anthropic advises authors to challenge every paragraph: does this justify its token cost?

That guidance matters because Claude may select from 100+ available skills using the description field, and concise skill examples run around 50 tokens while verbose ones hit 150, a 3× difference that adds up fast across an active agent session.

The practical takeaway: any "worth it" calculation has to account for both dimensions.

• 26.1% of marketplace skills contain at least one documented vulnerability
• Script-bundling skills are 2.12× riskier than instruction-only ones
• 5.2% show patterns strongly suggesting malicious intent
• Token costs compete across system prompt, history, and skill metadata
• Skill effectiveness varies by model, so testing on your target model matters

Teams adopting skills at scale need both security vetting and token-cost discipline before any performance claim becomes relevant. That's why independent benchmarking platforms now run every skill in a sandbox with and without the skill under identical conditions, security-scan and pin each version to a commit SHA, and publish results separately from any ratings. You can check current Lift scores and trust ratings for hundreds of skills free at skillzbench.com.

How to Judge a Skill Before You Install It: A Practical Checklist

Before installing any skill, ask one question: how do you know it actually helps? Most people never check. They read a description, hit install, and hope. Given what the evidence shows, that's a gamble.

The stakes are real. The first large-scale study of agent skills analyzed over 31,000 unique third-party skills and found 26.1% contain at least one vulnerability, spanning prompt injection, data exfiltration, and privilege escalation. Skills that bundle executable scripts are 2.12× more likely to be vulnerable than instruction-only ones. So a vetting process isn't paranoia; it's due diligence.

Here's a practical checklist to run any skill through before it touches your agent.

• Demand the methodology. Any claim that a skill "improves performance by X%" is only as good as the assertions used to grade the outputs, as independent reviewer Daniel Nest put it. Ask: was the skill tested against a no-skill baseline on the same model, under identical conditions? If you can't see that comparison, the number means nothing.
• Check the security posture. Was the skill scanned? Is it pinned to a verified commit so you know exactly what code you're running? Unvetted marketplace skills are precisely where the documented risks live.
• Test on your model, not a benchmark hero. Anthropic's own docs are blunt about this: skill effectiveness depends on the underlying model, and "what works perfectly for Opus might need more detail for Haiku." A result from a different model than yours tells you little.
• Justify the token cost. Anthropic calls the context window "a public good" and tells skill authors to challenge every paragraph: does it justify its token cost? A skill that adds marginal capability while bloating context can easily be net-negative.

That last point deserves a beat. Anthropic's examples show a concise skill description at roughly 50 tokens versus 150 for a verbose one, a 3× difference that compounds across every conversation and every other skill competing for the same window.

One more reason to systematize this: Anthropic notes that teams with evals can evaluate a new model and upgrade in days, while teams without them face weeks of testing. The same logic applies to skills. If you have a repeatable way to check whether a skill helps, you adopt the good ones fast and skip the rest. If you don't, every skill is a manual experiment.

This is exactly the gap SkillzBench was built to close. Its free directory benchmarks every skill in a sandbox with the same model, with and without the skill, and publishes the Lift in percentage points alongside a 0-100 trust score from security, quality, license, and maintenance signals. Every version is pinned to a commit SHA and security-scanned, and when upstream skills change, they get re-scanned and re-benchmarked. You can check the current results at skillzbench.com before you install anything.

Run every skill through this checklist and the "worth it" question stops being a matter of taste. It becomes a measurement.

Where to Find Actual Measured Results

The gap between "skills help" and "here's the proof" has been real for a while. That's changing, because independent benchmarking now exists and runs in the open.

SkillzBench publishes live Lift results for hundreds of skills at skillzbench.com. Every skill gets tested in a sandbox with the same model, with the skill and without it, and the difference comes back as Lift in percentage points. That's the with-and-without methodology Anthropic's own Skill Creator 2.0 uses, but published publicly and kept separate from any rating or revenue.

Why that separation matters: independent reviewer Daniel Nest argues that any "your skill improved by 40%" stat is only as good as the assertions used to grade it. When Lift is a raw measurement and ratings are a separate signal, you can see which one is doing the talking.

Alongside Lift, each skill carries a trust score from 0 to 100, built from five signals:

• Security: every version is pinned to a commit SHA and scanned for the vulnerability patterns the large-scale empirical study found in 26.1% of third-party skills
• Quality: instruction clarity and structure, judged against the same best practices Anthropic publishes for skill authors
• License: verified so you know what you can actually use
• Maintenance: whether the skill is still tracked upstream and re-scanned when it changes
• Community: adoption and maintenance signals from other users

Money never touches scores. The directory, the benchmarks, the alerts and the MCP connector are all free; SkillzBench earns revenue from advertising in and around its blog articles, and ad dollars don't influence listings or Lift.

When a skill looks worth equipping, the free MCP connector does the rest. You pick the skills and tools per agent (Claude Code, Codex, Cursor, Gemini CLI or a chat-app agent), the agent only sees what you chose, and you can swap loadouts per workflow. It's the practical answer to the token-cost problem Anthropic flags when it calls the context window "a public good": you pay for the skills you verified, and nothing else.

Head to skillzbench.com for current Lift numbers and trust scores on hundreds of skills. Skills, proven.