The Hidden Risks of awesome-lists: Unvetted Skills at Scale

The Numbers Nobody Reads Before Installing

When NVIDIA researchers analyzed 31,132 agent skills, they found that 26.1% contained at least one vulnerability and 5.2% showed likely malicious intent, according to NVIDIA's SkillSpector research. These are not projections or worst-case scenarios — they are measured findings from one of the largest skill datasets ever analyzed, spanning 71 vulnerability patterns across 17 categories.

A separate audit reached a similarly uncomfortable conclusion. A Snyk scan of 3,984 skills from ClawHub and skills.sh found that 13.4% carried at least one critical-level security issue, while 36.82% had at least one flaw at any severity. Two independent studies, two different corpora, one convergent message: roughly a quarter to a third of published skills carry real risk.

What makes these numbers worse in practice is how skills get distributed. Marketplaces at least run some scanning, however limited. Awesome-lists — curated GitHub link lists — typically apply no verification at all. Placement on a list usually reflects a maintainer's judgment or a community member's pull request, not a security review. And the trust signals people do rely on are trivially gameable:

• Install counts can be inflated with unauthenticated scripted requests — one proof-of-concept pushed a skill to several thousand installs before it was flagged.
• Upvotes, stars, and comments can be faked with bots and throwaway accounts.
• Marketplace scanning often runs only at creation or at a popularity threshold, missing later changes.
• Bulk updates can ship without diffs or version pinning, silently swapping what a skill does.

Orca Security, which documented those attack primitives, put it bluntly: the rigor applied to package managers and container registries needs to extend to skill distribution, and until then, every skill should be treated as untrusted code.

The scale of adoption amplifies the exposure. One skill catalog entry, "Find-Skills," reportedly surpassed 200,000 downloads within roughly three months of Anthropic's December 2025 Agent Skills announcement, per Mitiga's analysis. Distribution at that velocity, with vetting this thin, is a demonstrated supply chain risk — not a hypothetical one.

This is the gap SkillzBench is being built to close: independent verification that pins every skill version to a commit SHA, runs security scans and license checks, and publishes a 0–100 trust score computed from security, quality, and maintenance signals — with popularity kept out of the math entirely. Until that kind of verification layer becomes standard, the numbers above are the due diligence most installs never get.

Why Stars, Installs, and Upvotes Can't Protect You

If a skill has 50,000 downloads, hundreds of stars, and a glowing comment thread, it must be safe — right? That instinct is exactly what attackers are counting on, and security researchers have now demonstrated how cheap it is to fake.

Orca Security's proof-of-concept inflated a skill's install count "to several thousand" using nothing more than scripted, unauthenticated HTTP requests — one request per fake install. The skill was only flagged after it had already achieved widespread distribution. Meanwhile, Mitiga's research documented how bot upvotes, fake downloads, and fabricated comments can inflate a skill's apparent trustworthiness on skills.sh, the largest public skill catalog. The numbers that look like social proof are, in practice, marketing.

This matters because awesome-lists inherit these signals wholesale. A curated list typically surfaces skills based on stars, activity, and community chatter — all of which are trivially gameable. The curation adds a human eyeball at one moment in time, but nothing stops a maintainer from pushing a malicious update afterward, and nothing verifies the content beyond that initial glance.

The attacks in the wild confirm this isn't theoretical:

• Datadog Security Labs documented "Clawsights," a skill posing as a Claude Code leaderboard that instructed users to run gh auth token and exfiltrated the credential to an attacker-controlled domain via curl.
• Orca's proof-of-concepts achieved actual code execution on end-user systems, with logs showing unique affected machines across multiple networks.
• Mitiga's silent codebase-exfiltration PoC required only four user interactions and left the audit log completely empty.

And if your fallback is "the model will catch it," Datadog's testing undermines that too. Dynamic context commands execute during preprocessing — before the model sees the skill at all — so prompt-injection defenses never get a chance to intervene. Their version-comparison testing also showed inconsistent refusal behavior: one Claude model version refused a malicious skill, while a later version ran it without identifying the credential theft.

The lesson is that popularity measures attention, not safety. What's needed instead is verification: pinning every skill version to a commit SHA, running security scans continuously rather than once at creation, and benchmarking actual behavior in a sandbox. That's the model SkillzBench is building — independent testing where trust scores come from security, quality, and maintenance signals, and Lift is measured, never marketed. Until that layer exists everywhere, treat every skill as untrusted code, because that's exactly what it is.

Skills, proven. Join the SkillzBench waitlist to get independent benchmarks and security scores for the skills you actually use.

Why 'The Model Will Catch It' Fails

Most developers assume their AI agent acts as a last line of defense — that if a skill contains malicious instructions, the model will refuse them. Datadog Security Labs' research dismantles that assumption, and the implications are uncomfortable for anyone installing skills from an awesome-list.

The most damaging finding concerns dynamic context commands. Datadog found that commands using ! syntax execute during preprocessing — before the model sees the skill at all. Prompt injection defenses never intervene because there is nothing for them to evaluate; the code has already run by the time the model reads a single word (Datadog Security Labs).

Even when the model does see malicious instructions, refusal is inconsistent. In Datadog's testing, Opus 4.6 refused a malicious skill — but in later tests, Opus 4.7 ran the same skill without identifying the credential theft. Their researchers conclude that no model will ever be 100% resistant to prompt injection, meaning model-level guardrails cannot be your security boundary.

The attack surface extends beyond deliberate installs. Cloned repositories can carry skills via .claude/skills/ directories, including nested monorepo folders — Datadog notes that "a package can bring its own skills even if the repository root looks clean." A routine git clone can silently wire new instructions into your agent session, with skill-loading precedence rules (enterprise over personal, personal over project) determining what actually runs.

Mitiga Labs demonstrated how quiet this can be in practice. Their proof-of-concept exfiltrated an entire codebase with just 4 user interactions — and left the skill-audit.log completely empty (Mitiga Labs). Worse, when researchers asked the agent to make the malicious instructions "more silent," the agent itself co-designed a more effective attack.

The pattern across these findings is consistent:

• Malicious behavior can execute before the model has any chance to refuse it.
• Refusal behavior varies across model versions, so an update can silently change your risk profile.
• Skills can enter sessions through cloned repos, not just explicit installs.
• Attacks leave audit trails that show nothing unusual happened.

This is why "the model will catch it" fails as a strategy. Verification has to happen upstream of the agent session — which is the gap SkillzBench is building to fill, by security-scanning and version-pinning skills before they ever reach your agents.

The Minimum Bar: Pin, Scan, Re-Scan, Scope

If a skill can rewrite what your agent runs, "I trust the author" is not a security posture. The good news is that security researchers have converged on a clear minimum bar — and none of it involves stars, upvotes, or a curator's good taste.

First, pin everything. Orca Security's research found that marketplaces allow blind bulk updates with no diff or version pinning, meaning the skill you reviewed yesterday can silently become a different skill today. Pinning a skill to a specific commit is the only way to know the code you scanned is the code your agent loads.

Second, treat every skill as third-party code. Orca's guidance is blunt: "treat every skill as untrusted code, because that's exactly what it is." Cloud Security Partners' Mike McCabe echoes this, arguing skills deserve the same review process as any dependency — with a blast radius that is "arguably worse," since they shape agent behavior directly.

Third, scan continuously — and never trust a single scanner. OWASP's Agentic Skills Top 10 scanner integration guidance notes that no single scanner covers all ten risk categories; even NVIDIA's SkillSpector, the leading tool, addresses only seven. One-off scans are equally insufficient — skills change upstream, so re-scanning has to be a standing process, not an install-time event.

Fourth, scope what the agent can touch. OWASP's prescriptive guidance is least privilege with fail-closed defaults: grant agents the minimum tools a task requires and deny unknown tools outright.

The minimum bar, condensed:

• Pin every skill to a specific commit SHA — no blind updates
• Review each skill as untrusted third-party code before it loads
• Run continuous, multi-layer scanning across all OWASP AST categories
• Scope tools per agent, failing closed on anything unapproved

This is where independent verification infrastructure becomes essential rather than nice-to-have. SkillzBench is building exactly this workflow: every skill version pinned to a commit SHA, validated, license-checked, and security-scanned — then re-scanned and re-benchmarked whenever upstream changes, with notifications instead of silent drift. Per-agent loadouts enforce the least-privilege principle mechanically, so an agent only sees the skills and tools someone deliberately chose for it. Given that NVIDIA's large-scale analysis found 26.1% of 31,132 skills contain vulnerabilities, verification has to be infrastructure, not an afternoon of manual review. Skills, proven — that is the standard awesome-lists can't meet on their own.

Your Practical Skill-Hygiene Checklist

You don't need to wait for the ecosystem to mature to protect yourself — the security teams who found these problems already agree on what good skill hygiene looks like. Orca Security puts it bluntly: treat every skill as untrusted code, because that's exactly what it is.

Start by auditing what's actually loaded. Skills don't just arrive through explicit installs — Datadog found that cloned repositories can carry skills in .claude/skills/ directories, including nested monorepo folders, so "a package can bring its own skills even if the repository root looks clean." Check every session's loaded skills against what you intended.

Pin your versions. Orca's research documented blind bulk updates with no diff or version pinning as a core attack primitive — a skill you reviewed last month can silently change underneath you. Pinning each skill to a specific commit SHA means you always know exactly which instructions your agent is running, and any change is a deliberate decision rather than a surprise.

Read the SKILL.md before enabling anything. With NVIDIA's analysis finding 26.1% of 31,132 skills contain vulnerabilities and 5.2% showing likely malicious intent, a five-minute read is cheap insurance. Watch for shell commands, curl calls, and anything touching credentials — Datadog's in-the-wild "Clawsights" sample instructed users to run gh auth token and exfiltrated it via curl.

Your working checklist:

• Audit loaded skills in every agent session, including skills inherited from cloned repos
• Pin each skill to a specific commit SHA and review diffs before any update
• Read SKILL.md contents before enabling — flag shell commands, network calls, and credential access
• Scope each agent's tools to the minimum its task requires, per OWASP's guidance on least privilege and failing closed
• Never run bulk updates blind — popularity signals like install counts can be inflated with scripted requests

Doing all of this manually is exactly the gap SkillzBench is being built to close. The platform will verify skills against pinned commit SHAs, run security and license scans, benchmark every skill in a sandbox to publish its Lift, and let you define per-agent loadouts so each agent only sees the skills and tools you explicitly chose. If you want independent verification instead of trust-by-star-count, join the SkillzBench waitlist for early access as the platform launches — because skills should be proven, not presumed.