What Makes a Skill High Quality?

The Quality Gap: Why Most Public Skills Fail

The skills ecosystem has a supply problem that looks like success from a distance. SkillsMP indexes roughly 1.9 million public skills scraped from GitHub, and daily submissions to ClawHub jumped from under 50 in mid-January to over 500 by early February, a 10x surge in weeks. Yet the average public skill scores only 6.2 out of 12 on quality, and the gap between catalog size and actual utility is widening, not closing.

Volume has outpaced verification. Publishing a skill requires only a SKILL.md file and a one-week-old GitHub account: no signing, no review, no sandbox by default. The result is a marketplace where helpful procedural guides sit alongside risky clutter, and users have no built-in way to tell them apart.

The security dimension sharpens the problem. A Snyk audit of 3,984 skills found 36.82% contained at least one security issue and 13.4% carried at least one CRITICAL flaw. Skills inherit the full permissions of the agent they extend: shell access, file-system read/write, credentials, persistent memory. Snyk researchers describe the current state as resembling the early days of npm and PyPI, except with unprecedented access to sensitive systems.

Several structural problems explain why the quality gap persists:

• No quality floor at publish time. A SKILL.md and a week-old account are the only gate, so low-quality and malicious entries enter the same index as vetted ones.
• Self-generated skills add no value. Research from the SkillsBench team found that models cannot reliably author the procedural knowledge they benefit from consuming, so LLM-authored skills tend to add noise rather than lift.
• Monolithic structure hurts performance. Skills that try to cover everything in one document reduced agent performance by 2.9 points, while focused 2–3 module skills improved it by 18.6 points.
• Popularity signals are unreliable. Download counts and star totals conflate helpful skills with well-marketed ones, giving users no empirical signal about whether a skill actually helps.

The practical consequence is that anyone assembling an agent loadout faces a needle-in-a-haystack problem. The skills that demonstrably improve performance exist, but they are buried in a catalog where the median entry is mediocre and a meaningful minority is actively dangerous. Without independent verification, security scanning, and A/B benchmark data, users are choosing blind.

This is exactly the gap that platforms like SkillzBench are building to close: continuously crawled and deduplicated directories, pinned version checks, security scans, and sandboxed before-and-after benchmarks that publish the lift in percentage points. When the directory itself is free and the scoring is never influenced by revenue, the quality gap becomes navigable rather than permanent.

Anatomy of a High-Value Skill: Scope, Security, and Authorship

Not all skills are created equal — and the research now tells us, with unusual precision, exactly which structural attributes separate the skills that lift agent performance from the ones that drag it down. Three attributes matter most: scope, authorship, and security.

Scope: narrow beats comprehensive. Focused skills containing 2–3 targeted modules improved agent performance by 18.6 points, while monolithic "put everything in one document" skills actually reduced performance by 2.9 points, according to ecosystem research. That's a spread of more than 21.5 points between the two approaches.

The reason is attention. As NVIDIA's evaluation work explains, every skill in an environment competes for the agent's attention. A bloated skill floods the context window with marginally relevant instructions, while a tightly scoped one delivers exactly the knowledge needed for a task class. For builders composing skill libraries, this means selecting several targeted skills rather than one do-everything package.

Authorship: humans write what models can't. The SkillsBench research is blunt on this point: "models cannot reliably author the procedural knowledge they benefit from consuming." LLM-generated skill content was judged "most of the time of low quality," and self-generated skills provided no benefit on average.

The distinction is procedural versus declarative. A quality skill teaches an agent how to approach a task — workflows, decision points, edge cases — rather than simply stating what to output. Models already know facts; what they lack is the hands-on, hard-won procedural judgment that only a human practitioner can articulate.

Security: the non-negotiable prerequisite. None of the above matters if the skill is dangerous. Snyk's ToxicSkills audit of 3,984 skills found that 36.82% contained at least one security issue, and 13.4% carried at least one CRITICAL flaw, as documented in their security research. Worse, publishing a skill requires only a SKILL.md file and a week-old GitHub account — no signing, no review, no sandbox by default.

Skills inherit the full permissions of the agent they extend: shell access, file system read/write, credentials, and persistent memory. Snyk compares the ecosystem's current state to early npm and PyPI, before security became a first-class concern. A high-value skill, then, is one that has been:

• Scoped narrowly to 2–3 modules covering a defined task class
• Authored by a human with genuine procedural expertise
• Written as how-to guidance, not declarative answers
• Scanned for prompt injection, malicious code, and hardcoded secrets

Security cleanliness comes first — a skill that fails scanning is disqualified regardless of how well it performs. SkillzBench is being built around exactly this principle: every skill in its directory will be security-scanned and quality-judged before it earns a trust score, so you never have to gamble on an unvetted package.

Measuring Lift: The Empirical Standard for Skill Value

Popularity tells you nothing about whether a skill actually works. The only honest test is dynamic: run the agent with the skill and without it, on the same tasks, and measure the difference.

That difference is what benchmarkers call Skill Lift. In a controlled sandbox study spanning 7,308 trajectories across seven agent-model configurations, curated skills raised average pass rates by 16.2 percentage points, according to the SkillsBench research paper. Gains were not uniform: domain-level improvement ranged from +4.5 points in software engineering to +51.9 points in healthcare.

NVIDIA's SkillEvaluator, testing more than 300 verified skills, reported an average Skill Lift of 41 points on correctness — with correctness scores rising from 46 to 87 once skills were added, per NVIDIA's evaluation methodology.

Two caveats matter for anyone reading a benchmark number:

• Skill value is model-dependent. Models adhere to encoded instructions to widely varying degrees, so the same skill can produce very different gains across configurations.
• Harness differences are small by comparison — Claude Code and Codex differ by roughly 5 points on average — while per-product lift ranges from about +2 to +46 points.
• Skills can also hurt. Sixteen of 84 tasks in the SkillsBench study showed negative deltas, and an irrelevant skill loading into context can actively degrade performance.
• Token savings are not guaranteed; some skills cut usage sharply while others increase it.

This is precisely why SkillzBench treats Lift as a separate, published number rather than folding it into a single score. The platform is being built to pin every skill version to a commit SHA, run it in a sandbox with and without the skill under the same model, and report the resulting lift in percentage points — kept apart from trust ratings, with money never touching either.

Skills, proven. Join the waitlist at skillzbench.com to follow the benchmarks as they publish.

Practical Framework for Evaluating and Deploying Skills

Knowing that a quarter of published skills carry vulnerabilities changes how you should vet one before it ever touches your agent. A skill inherits everything your agent can reach—shell access, file systems, credentials—so installing an unscanned skill is closer to running an unreviewed dependency than reading a document.

Before deploying any skill, run through a short checklist grounded in what the research consistently shows:

• Verify a security scan happened. Snyk's ToxicSkills audit found 36.82% of scanned skills contained at least one security issue, and 13.4% had a critical flaw. Publishing a skill requires only a SKILL.md and a week-old GitHub account—no review, no sandbox.
• Check for focused scope. Skills with 2–3 targeted modules improved performance by 18.6 points, while monolithic "do-everything" skills reduced it by 2.9 points, according to the Agentman ecosystem report. Prefer several narrow skills over one comprehensive one.
• Confirm human authorship and procedural content. The SkillsBench research found self-generated skills provide no benefit on average—"models cannot reliably author the procedural knowledge they benefit from consuming." A good skill teaches how to approach a task class, not what to output.
• Test on your actual setup. Skill value is model-dependent: per-product lift ranged from +2 to +46 points while harness differences averaged only about 5 points, so cross-configuration results may not transfer to your agent.

That last item is the hard part. Most teams have no way to run a controlled with-versus-without comparison, which is why raw availability and proven utility remain two different things. The average public skill scores just 6.2 out of 12 on quality, and 16 of 84 benchmark tasks actually showed negative deltas when a skill was added.

This is the gap SkillzBench is being built to close. The free directory continuously crawls and dedupes skills from GitHub, pins every version to a commit SHA, and runs security scans, license checks, and quality judgments before anything is listed. Each skill gets benchmarked in a sandbox with the same model, tested with and without the skill, and the resulting Lift is published in percentage points—separately from the 0–100 trust score, and never influenced by money. An upstream change triggers a re-scan and re-benchmark, so a skill that was safe in January doesn't silently become a liability in March.

The takeaway for anyone deploying skills today: treat security scanning, focused scope, and human authorship as non-negotiable gates, and demand measured lift rather than download counts as the proof of value. When SkillzBench launches, that verification work will be done for you—join the waitlist to be first in line when skills, proven, are ready to browse.