Why Security Scanning Alone Is Not Enough for Skills

The 26% Problem: Skills Are Vulnerable and Scanning Keeps Missing It

Every day, thousands of developers install AI agent skills believing a security scan has cleared them. The data says otherwise: more than a quarter of those skills carry vulnerabilities the scan never caught — and some are actively malicious.

The numbers are hard to ignore. A large-scale academic study analyzing 31,132 unique skills across two major marketplaces found that 26.1% contain at least one vulnerability, spanning prompt injection, data exfiltration, privilege escalation, and supply chain risks. Snyk's independent audit of 3,984 skills from ClawHub and skills.sh was even starker: 36.82% had at least one security issue, and 13.4% carried a critical one.

The most sobering evidence comes from a real-world incident. As documented by CSOOnline, a malicious skill called "brand-landingpage" passed security checks from Cisco, Nvidia, and skills.sh scanners — all returned "safe" verdicts — and reached more than 26,000 users. The trick was simple: the payload sat behind a mutable external URL that was changed after distribution, so the submitted package looked clean.

This is the structural problem, not a one-off failure:

• Skills can instruct agents to perform malicious actions in natural language, bypassing code-level detection entirely.
• Traditional SAST tools don't scan markdown, and traditional SCA tools don't model skill dependencies, as Fortinet notes in its analysis of the skills supply chain.
• Point-in-time scans fail when attack logic lives on remote infrastructure, modifiable after publication.
• Publishing requires only a SKILL.md file and a week-old GitHub account — no code signing, no review, no sandboxing by default.

Even Anthropic, whose scanning tooling many teams rely on, is candid in its official documentation: a pass result "isn't a guarantee that a skill is safe in every respect," and it won't catch a skill that behaves in unintended ways without being malicious.

That last admission points to a second, quieter gap: scanning tells you nothing about whether a skill actually helps. It cannot tell you if a skill degrades your agent's output quality, wastes tokens, or subtly changes behavior. This is why SkillzBench pairs security scanning with sandboxed benchmarking — testing every skill with the same model, with and without it, to measure real Lift — and pins every version to a commit SHA so a "safe" verdict can't silently rot as upstream code changes.

Scanning is necessary. It is not sufficient. The 26% problem isn't that scanners are bad — it's that they're being asked to do a job they were never designed to finish.

Why Scanners Are Structurally Blind: Natural-Language Attacks, Mutable Payloads, and Vendor Admissions

If a skill can tell an agent to do something malicious in plain English, no code scanner will ever see it. That's not a hypothetical limitation — it's the structural blind spot at the heart of every point-in-time security scan on the market today.

The first problem is that skills are written in natural language. A large-scale analysis of 31,132 agent skills found that skills can instruct agents to perform malicious actions in plain text, bypassing code-level detection entirely. Snyk researchers put it bluntly: prompt injection has "no analog" in traditional package security. Fortinet drives the point home — traditional SAST tools don't scan markdown, and traditional SCA tools don't model skill dependencies. The attack surface lives in a format the existing tooling was never built to inspect.

The second problem is that what a scanner examines isn't necessarily what runs later. A documented real-world experiment showed a malicious skill called "brand-landingpage" pass checks from Cisco, Nvidia, and skills.sh — all three returned "safe" — and reach 26,000+ users. The payload wasn't in the submitted package at all; it sat behind a mutable external URL that was changed after distribution. Snyk independently documented the same pattern, flagging unverifiable dependencies like curl | source that let attackers alter skill behavior post-publication. A point-in-time scan captures a snapshot; the attack lives in the delta.

Third, even a perfectly clean skill can become a threat vector. Snyk's "Insecure by Design" finding showed that a legitimate, well-reviewed skill fetching third-party content gets compromised when an attacker poisons that public source. As Snyk wrote: "The skill author did nothing wrong. The user installed a popular, well-reviewed skill. Yet the agent is compromised." Unit 42 telemetry confirms this is actively weaponized in the wild, with 22 distinct indirect prompt injection payload techniques observed — one single webpage contained 24 separate injection attempts.

• Natural-language prompt injection evades code-based detection entirely.
• Mutable external URLs let payloads change after a scan has passed.
• Indirect injection compromises clean skills through poisoned third-party sources.
• Vendor documentation itself admits a passing scan is not a safety guarantee.

That last point deserves emphasis because it comes from the vendor running the scanner. Anthropic's own documentation states that a pass result "isn't a guarantee that a skill is safe in every respect," and that scanning won't catch a skill that behaves in ways you didn't intend without being malicious. Anthropic also documents explicit coverage gaps — skills installed before scanning was enabled, skills shared via connected MCP servers, and MCP servers and hooks aren't scanned at this time.

This is why SkillzBench treats scanning as one pillar of verification, not the verdict itself. Every skill version is pinned to a commit SHA and continuously re-scanned as upstream changes land, and behavioral benchmarking — testing each skill in a sandbox with and without it — catches the unintended misbehavior that static scans structurally cannot. Security experts converge on the same conclusion: skills are living supply-chain dependencies requiring continuous validation and least-privilege runtime controls, not a one-time green checkmark.

The Expert Consensus: Treat Skills as Living Supply-Chain Dependencies

When a malicious skill passed scanners from Cisco, Nvidia, and skills.sh and still reached 26,000+ users, the security community took notice. The lesson, echoed by researchers and practitioners alike, is that skills need to be governed like any other third-party software dependency — not scanned once and forgotten.

Security researcher Devashri Datta put it bluntly in reporting on the incident: treating skills as mere text is a fundamental architectural misunderstanding. They are executable instruction bundles that must be governed with the same rigor as open-source packages. Keith Prabhu, founder of Confidis, agrees, calling skills "living third-party dependencies" and arguing that a one-time scan is no longer sufficient — enterprises need continuous validation and strict runtime controls.

The evidence backs them up. The payload in that breach sat behind a mutable external URL that was changed after distribution, defeating every point-in-time check. Snyk researchers documented the same pattern with unverifiable dependencies like curl | source, which let attackers alter skill behavior post-publication.

The emerging consensus converges on four practices:

• Continuous validation, not one-time scans. Because payloads can change after publication, skills must be re-scanned as upstream content evolves — a skill that was safe at install time may not stay safe.
• Version pinning to commit SHAs. Pinning every skill version to an immutable commit makes external dependencies verifiable and closes the mutable-URL loophole that fooled mainstream scanners.
• Capability-based permissions and least privilege. The authors of the large-scale skills study explicitly call for capability-based permission systems and mandatory security vetting before this attack vector is exploited further.
• Behavioral testing beyond security. Anthropic's own documentation admits a passing scan "won't catch a skill that behaves in ways you didn't intend without being malicious" — so testing how a skill actually performs, in a sandbox, is essential.

That last point matters more than most teams realize. Scanning answers "is this skill malicious?" but not "does this skill actually help, or quietly misbehave?" With 26.1% of 31,132 analyzed skills containing at least one vulnerability, and 36.82% of Snyk's audited sample carrying at least one security issue, the gap between "passed the scan" and "safe and useful" is wide.

This is the philosophy behind SkillzBench, which is building continuous verification — pinning each skill version to a commit SHA, re-scanning and re-benchmarking as upstream content changes — alongside per-agent loadouts where an agent only sees the skills and tools a user explicitly enabled. Behavioral benchmarking measures each skill's Lift separately from security ratings, catching the unintended misbehavior that scans structurally cannot.

The message from every credible source is the same: a scan result is a data point, not a guarantee. Treat skills like the living supply-chain dependencies they are.

From Scan to Proof: How Continuous Verification, Benchmarking, and Per-Agent Control Close the Gap

A malicious skill passed every scanner it faced — Cisco, Nvidia, and skills.sh all returned "safe" — and still reached 26,000+ users, because its payload sat behind a mutable external URL changed after distribution. That case, documented by CSOOnline, makes the expert consensus concrete: scanning must become one step in a continuous workflow, not the whole workflow.

That workflow starts with pinning every skill to an exact version. Treating skills as living supply-chain dependencies — the framing security researchers like Confidis founder Keith Prabhu advocate — means recording the specific commit SHA you validated, not "latest." When the upstream changes, yesterday's clean scan is void, and the skill must be re-scanned and re-judged.

But even a clean scan says nothing about whether a skill actually helps. Anthropic's own documentation admits a pass result "isn't a guarantee that a skill is safe in every respect" and won't catch behavior that is unintended rather than malicious. The only way to know what a skill really does is to run it: the same model, in a sandbox, on the same tasks, with and without the skill installed. The difference — measured in percentage points — is the skill's Lift. Behavioral testing catches exactly what static scanning structurally cannot.

The third pillar is least privilege at the agent level. The arXiv skills study concludes there is "an urgent need for capability-based permission systems," and with 26.1% of 31,132 analyzed skills carrying at least one vulnerability, blanket access is indefensible. In practice, that means each agent gets a scoped loadout:

• Toggle on only the skills a given agent needs for its job.
• Restrict tool access per agent, so a writing agent never sees shell tools.
• Enforce organizational allowlists so nothing unvetted reaches production agents.
• Audit who enabled what, and when, across every agent in the fleet.

Finally, the loop closes with monitoring: track upstream changes, re-scan, re-benchmark, and notify when something shifts. This is the full pipeline SkillzBench is building across its Discover, Verify, Benchmark, Equip, and Stay Current pillars — pinning each version to a commit SHA, publishing Lift separately from trust scores, and gating agent visibility through per-agent loadouts served over a single MCP connection.

Scanning answers "is this file clean today?" Verification, benchmarking, and control answer the questions that actually matter: "is this exact version safe, does it measurably help, and can this agent even see it?" That is the difference between a snapshot and a system.

Your Action Plan: What Teams Should Do With Skills Today

Knowing that roughly a quarter of installed skills carry at least one vulnerability changes how you should spend your next hour. The good news: a focused set of actions, taken today, closes most of the gaps that scanning leaves open.

Start with an inventory. List every skill installed across your agents, along with its version and source. Point-in-time scans can't protect you from what you don't know you're running — and as one documented case showed, a single malicious skill can silently reach 26,000+ users after passing every scanner it faced.

Prefer instruction-only skills. The large-scale empirical study of 31,132 skills found that script-bearing skills are 2.12× more likely to contain vulnerabilities than instruction-only ones. If a skill bundles executable code, treat it as high-risk until proven otherwise.

Cut unverifiable external dependencies. Snyk's audit found unverifiable dependencies in 21% of malicious samples — patterns like curl | source that let attackers change skill behavior after publication. If a skill fetches and executes remote content, remove it or pin and sandbox it.

Apply least privilege per agent. Anthropic's own guidance is blunt: a pass result "isn't a guarantee that a skill is safe in every respect," and users should choose what they add carefully. Concretely, that means:

• Toggle tools and skills per agent so each agent only sees what its task requires
• Pin skill versions to specific commits so upstream changes can't drift in silently
• Re-scan and re-evaluate whenever a skill updates, not just at install time
• Restrict network access for agents running skills that fetch third-party content

Finally, close the trust gap that scanning can't: whether a skill actually helps. Anthropic admits scanning won't catch skills that behave in unintended ways, and no tool today tells you which skills improve agent performance versus degrade it. SkillzBench is being built to answer exactly that — independent verification, sandboxed benchmarking that measures the Lift each skill adds, and per-agent loadouts that enforce least privilege by design. The product is pre-launch, but the waitlist at skillzbench.com is live, and early access goes to those who join first. Skills, proven — that's the standard your agents deserve.