The Interview: What 100 Skill Repos Taught Us About Maintenance

The Ungoverned Gold Rush: Why Agent Skills Are a Maintenance Blind Spot

The agent-skills gold rush happened in weeks, not years — and nobody checked the door on the way in. Five major marketplaces now index a combined 900,000+ skills, with daily submissions jumping from under 50 in mid-January 2026 to over 500 by early February — a 10x surge in a matter of weeks, according to Snyk's audit of the ClawHub ecosystem.

What makes this growth dangerous isn't just the pace. It's that publishing a skill requires only a SKILL.md Markdown file and a GitHub account one week old. As Snyk researchers put it: "No code signing. No security review. No sandbox by default." The ecosystem, they argue, has a supply chain problem that mirrors early npm and PyPI — except with unprecedented access to credentials, file systems, and APIs.

The numbers back that warning up. An arXiv study of 31,132 unique skills found 26.1% carry at least one vulnerability across prompt injection, data exfiltration, privilege escalation, and supply chain risks. Snyk's separate scan of 3,984 skills found 36.82% with at least one security issue, and 13.4% containing critical flaws — including 76 confirmed malicious payloads, 8 of which were still live at publication.

Here's the governance gap that makes those numbers sting:

• SKILL.md files get reviewed like READMEs. Because they're Markdown, pull request reviewers skim them as documentation — yet a skill can pre-approve its own tools via allowed-tools frontmatter without any permission prompt, as StepSecurity notes.
• Skills inherit full agent privileges: shell access, file system read/write, and credentials — the same access the agent itself holds.
• Merging a skill distributes it. Everyone who clones the repository gets it, silently.

The result is a category of software that behaves like code but is treated like documentation. A malicious skill doesn't need to exploit a vulnerability at all — it just needs a convincing English sentence that the agent follows. And skills bundling executable scripts are 2.12× more likely to contain vulnerabilities than instruction-only skills, per the arXiv analysis.

This is the blind spot SkillzBench is being built to close: pinning every skill version to a commit SHA, security-scanning, and sandbox-testing before anything reaches your agent — the vetting layer the marketplaces skipped entirely.

What the Evidence Actually Says: Five Hard Maintenance Lessons From the Skill Ecosystem

Behind the 900,000+ skills now indexed across major marketplaces sits an uncomfortable truth: almost nobody is maintaining them the way software demands. Two independent audits and a large-scale abandonment study converge on the same conclusion — skills need the same governance discipline as any production dependency.

Lesson one: executable scripts multiply risk. A multi-institutional audit of 31,132 unique skills found that skills bundling executable scripts are 2.12× more likely to contain vulnerabilities than instruction-only skills. The same study pegged the overall vulnerability rate at 26.1%, with 5.2% showing high-severity patterns strongly suggesting malicious intent.

Lesson two: curation beats bulk. Practitioner analysis shows that installing "everything" can consume 100K+ tokens per turn, while a curated setup of roughly 20 high-value skills retains full functionality at a fraction of the cost. Profile-based installation cuts token overhead by 50–80%. More skills is not more capability — it's more context tax.

Lesson three: "shadow skills" recreate dependency chaos. Without a registry, skills proliferate as ad hoc scripts invisible to platform teams, a problem JFrog calls "directly analogous to the pre-Artifactory era of software dependency management." Version confusion follows: two agents, two copies of the same runbook, two different answers.

Lesson four: abandonment is the norm, and stars won't save you. According to OSS abandonment research, 91% of audited open-source components show no clear maintenance in two years, and surface metrics like stars, commits, and PRs are volatile and manipulable. A skill with 40K stars may still be a security liability.

Lesson five: a clean scan is not a safety certificate. Snyk's audit of 3,984 skills found 13.4% with critical issues, and pattern-matching scanners have marked deliberately malicious skills as "CLEAN." As JFrog puts it, static scanning cannot catch subtle natural-language manipulation — a malicious skill only needs "a convincing English sentence."

The practical takeaways for any team adopting skills:

• Treat every SKILL.md change with CI/CD-level scrutiny — a skill that pre-approves Bash deserves the same review as a workflow change.
• Pin versions; never let agents fetch whatever is at the top of a marketplace this week.
• Combine scanning with sandboxed behavioral testing, not pattern matching alone.
• Track upstream changes continuously, because abandonment is the base rate, not the exception.

This is exactly the discipline SkillzBench is being built to operationalize — pinning each version to a commit SHA, re-scanning and re-benchmarking as upstream changes land, and letting you curate per-agent loadouts so agents see only what you chose. Maintenance, not installation volume, is what makes skills safe.

The Maintenance Playbook: Vetting, Versioning, and Curation Done Right

The good news buried in all this research: none of it requires new invention. Every problem we have seen — vulnerable skills, shadow versions, abandoned repos, token bloat — already has a proven answer from traditional software engineering. The playbook is simply applying it to SKILL.md files.

Start by treating skill selection as a vetting exercise, not a download. With one large-scale audit finding that 26.1% of skills carry at least one vulnerability, and Snyk's scan of 3,984 skills flagging 13.4% with critical issues, "install and hope" is not a strategy. Pin every skill version to a commit SHA, so what you tested is what you run.

Second, never trust a single scanner. Pattern-matching tools have marked deliberately malicious skills as "CLEAN," and as JFrog puts it, a clean scan lowers risk but does not prove safety. Combine automated scanning with sandboxed behavioral testing — run the same task with and without the skill and watch what it actually does. And review SKILL.md changes in pull requests with CI/CD-level scrutiny: because skills can pre-approve their own tools via frontmatter, a skill edit is a permissions change, not a README tweak.

Third, curate aggressively and watch for abandonment:

• Keep loadouts lean — bulk installs can consume 100K+ tokens per turn, while ~20 curated skills retain most of the value.
• Ignore surface metrics like stars and commits; research on 115,466 GitHub repositories shows behavioral signals predict abandonment far better.
• Track upstream changes continuously, since 91% of audited OSS components show no clear maintenance within two years.
• Re-scan and re-test whenever a skill author pushes an update — yesterday's clean skill can become today's risk.

This is exactly the discipline SkillzBench is being built to operationalize. Every listed version will be pinned to a commit SHA, scanned, and benchmarked in a sandbox with the same model — with and without the skill — so users see the Lift in percentage points rather than marketing claims. Per-agent loadouts, delivered through a free MCP connector, keep each agent seeing only what someone deliberately chose for it.

The core shift is mental: a skill is not a snippet of documentation. It is a dependency with shell access, file system privileges, and your credentials. Vet it, pin it, test it, and watch it — the same way your team already treats every package in the lockfile.

Putting It Into Practice: A Team Checklist for Skill Maintenance

Knowing that 26.1% of skills carry at least one vulnerability — and that Snyk found 13.4% with critical flaws — turns maintenance from a nice-to-have into a daily practice. Here is a checklist your team can run this week, built directly from what the audits found.

Start with an audit of what's already installed. Skills can pre-approve their own tools through allowed-tools frontmatter, bypassing permission prompts entirely. As StepSecurity observes, a skill that pre-approves Bash deserves the same scrutiny as a CI/CD workflow change — so grep every installed SKILL.md for tool pre-approvals you didn't sign off on.

Review skills in pull requests like code. SKILL.md files "tend to get reviewed like a README," yet merging a skill distributes it to everyone who clones the repo. The stakes are measurable: one large-scale audit found skills bundling executable scripts are 2.12× more likely to contain vulnerabilities than instruction-only skills.

Curate per agent instead of bulk-installing. Bulk installs can consume 100K+ tokens per turn, while a curated ~20-skill loadout retains most of the value at a fraction of the cost — practitioner analysis shows profile-based curation cuts token overhead by 50–80%.

Set up change tracking and re-scan alerts. Surface metrics like stars and commits are unreliable health signals, and OSS abandonment research found 91% of audited components showed no clear maintenance in two years. Track upstream commits on every skill you rely on, and re-scan whenever they change — remembering that a clean scan lowers risk but doesn't prove safety.

Your working checklist:

• Audit installed skills for pre-approved `allowed-tools` frontmatter and remove anything you didn't explicitly authorize.
• Require PR review for SKILL.md changes with the same rigor as workflow or pipeline changes.
• Build a curated loadout per agent — roughly 20 high-value skills — rather than installing everything.
• Pin each skill to a specific commit SHA so you always know which version is running.
• Monitor upstream repos for changes and trigger a re-scan on every update.

This is exactly the discipline SkillzBench is being built to automate end to end: a free directory that pins every skill version to a commit SHA and security-scans it, a free MCP connector serving curated per-agent loadouts, and maintenance alerts that track upstream changes, re-scan, and notify you when a skill you use moves. It's currently in pre-launch with the waitlist open — and everything on the platform, from benchmarks to alerts, will be free.