Skill Hygiene: A Quarterly Maintenance Routine

Why a One-Time Security Check Isn't Enough: The Silent Update Problem

You audited every skill before installing it. The scan came back clean, the permissions looked reasonable, and you moved on with the quiet confidence of someone who has done their due diligence. Here's the uncomfortable truth: that verdict has an expiration date, and it may have already passed.

Security researchers at Orca demonstrated an attack they call Delayed Weaponization via Update. An attacker publishes a genuinely benign skill, waits for users to install and trust it, then modifies it days or weeks later to deliver a malicious payload — landing exactly when nobody is re-checking. The skill you vetted in January is not necessarily the skill running on your agent today.

The update mechanism itself is part of the problem. Orca's research documented several supply-chain weaknesses that make "set it and forget it" dangerous:

• Silent skill overrides on name collisions — an installed skill can be replaced by a same-named one from a different repository without any confirmation, diff, or warning
• Blind bulk-update commands that refresh every installed skill at once, with no per-skill review or version pinning
• Non-deterministic scanning that only runs at creation or popularity thresholds, letting a skill slip past review entirely

Even your original scan may have been less reassuring than it seemed. Cisco's benchmarks found that static rules alone catch only about 7.7% of malicious skills at high severity — and their team's caveat is worth memorizing: "No findings ≠ no risk." Regex-based scanners are trivially bypassed with tricks as simple as c${u}rl parameter expansion or plain natural-language instructions, as Snyk researchers documented.

The scale of what's already out there makes recurring checks non-negotiable. A Unit 42 analysis of nearly 50,000 registry skills found that 80% showed at least one mismatch between what they claim to do and what they actually do — and 5% carried multi-stage attack chains like file reads chained into network exfiltration. Separately, an academic study found roughly 26% of skills carry at least one vulnerability. These measure different things, but they point the same direction: the ecosystem resembles early npm or Docker, before security became a first-class concern.

So how often should you re-verify? No study prescribes a cadence, but the attack patterns do. Delayed weaponization plays out over days or weeks; silent overrides can happen any time an update runs. That makes a quarterly review a practical floor — inventory every installed skill, remove anything unrecognized, review upstream changes before updating, and re-scan with intent-aware tools. Continuous monitoring is the ideal, and some researchers argue periodic scanning is never enough.

This is the gap SkillzBench is being built to close: every skill version pinned to a commit SHA, re-scanned and re-benchmarked when upstream changes, with alerts when something shifts. If you'd rather not track skill drift by hand, join the waitlist — the directory, scanning, and alerts will all be free.

What the Research Says About Risk: The Numbers Behind a Maintenance Routine

If you install agent skills and never look at them again, the odds are already working against you. Multiple independent studies have measured the skill ecosystem from different angles, and while their numbers differ, they all point the same direction: a meaningful slice of what you're installing is broken, mismatched, or hostile.

An academic analysis of over 31,000 unique skills found that 26.1% carry at least one vulnerability, with data exfiltration (13.3%) and privilege escalation (11.8%) the most prevalent categories. A separate ToxicSkills audit of 3,984 registry skills measured the problem at higher severity: 13.4% contained critical security issues, 36.82% had at least one flaw at any severity, and 76 skills carried confirmed malicious payloads. These figures measure different things — vulnerabilities versus critical issues versus outright malice — so they complement rather than contradict each other.

The third number is the strangest. Unit 42's behavioral verification across 49,943 registry skills found that 80% of skills do something different from what they declare — mismatches between the SKILL.md's stated purpose and the skill's actual behavior, 250,706 deviations in total. Before you panic, note what most of those turned out to be: 81.1% of classified deviations were developer oversight, plain sloppiness. Only 18.9% showed adversarial intent.

But that adversarial slice is concentrated in the capabilities that hurt most:

• 60% of adversarial deviations were data theft or espionage — the silent credential exfiltration pattern.
• Instruction-manipulation capabilities were 96% adversarial, versus just 10% for file-system capabilities.
• Two dominant attack chains — credential exfiltration and instruction hijacking — cover 88% of multi-stage attacks.
• 5% of skills carried multi-stage attack chains that researchers flagged as mandatory-review.

The concentration matters because it tells you where to spend review effort. A skill that reads files is usually just badly written. A skill that touches credentials or rewrites the agent's own instructions deserves a hard look every time — and Unit 42's data says that hard look pays off, since those categories carry most of the actual malice.

The final number is the uncomfortable one for anyone relying on a quick keyword scan. Cisco's benchmarks on held-out malicious skills show static rules alone catch only 7.7% of malicious skills at HIGH severity — rising to 66.7% recall when paired with an LLM judge. In other words, the tool most people reach for first misses more than nine in ten high-severity malicious skills. Cisco's own caveat is blunt: no findings does not mean no risk.

Taken together, these numbers justify a recurring routine rather than a one-time check. Roughly a quarter of skills are vulnerable, one in eight has critical issues, four in five behave differently than declared, and yesterday's clean scan catches almost nothing today. That's why SkillzBench is building continuous re-scanning and re-benchmarking into its free directory — pinning every skill version to a commit SHA and re-verifying when upstream changes — so a skill you vetted in January gets re-checked in April without you remembering to ask.

Skills, proven. Join the SkillzBench waitlist and get the numbers on the skills you choose — free, always.

The Quarterly Skill Hygiene Checklist: Five Steps That Map to the Research

A skill you approved in January may not be the skill running in March. Security researchers at Orca Security demonstrated that skills can be silently modified after installation — their "Delayed Weaponization via Update" attack pushed a benign skill, then made it malicious days or weeks later, achieving real code execution on end-user machines. One-time vetting cannot catch that. A recurring routine can.

The evidence base is stark enough to justify putting this on the calendar. A multi-institutional study of more than 31,000 unique skills found that 26.1% contain at least one vulnerability, with data exfiltration (13.3%) and privilege escalation (11.8%) the most prevalent categories. Here is a five-step quarterly routine, each step mapped directly to what the research shows.

• Step 1 — Reconcile your inventory. List every skill installed across every agent, then remove anything you don't recognize. Orca's research found installed skills can be overridden on name collisions with no confirmation, diff, or warning — so unrecognized entries deserve hostile scrutiny, not benefit of the doubt.
• Step 2 — Re-scan with intent-aware tools. Keyword scanners are trivially bypassed by obfuscation like c${u}rl or plain natural-language instructions. Cisco's measured benchmarks show static rules alone catch only 7.7% of malicious skills at HIGH severity, rising to 66.7% when paired with an LLM judge. Their caveat applies every quarter: no findings ≠ no risk.
• Step 3 — Pin versions and review diffs before updating. Never run blind bulk updates, which refresh everything at once with no per-skill review. Pin to specific commits where possible and check repositories for recent changes first.
• Step 4 — Delete unused skills. Every installed skill inherits the agent's full permissions — shell access, file system, credentials, memory — so a dormant skill is an unmonitored attack surface, not a harmless leftover.
• Step 5 — Review execution contexts for least privilege. Configure each agent to see only the skills and tools you explicitly grant it, and tier your scrutiny by capability category.

That last tiering matters because adversarial intent is not evenly distributed. Unit 42's behavioral analysis across 49,943 registry skills found instruction-manipulation capabilities were 96% adversarial, versus just 10% for file-system capabilities — so a single review threshold either over-blocks routine skills or under-reviews dangerous ones. Give credential and instruction-level capabilities critical severity every time.

Treat quarterly as a floor, not a ceiling — Snyk's researchers advocate continuous scanning, and the supply chain doesn't respect your calendar. SkillzBench is building exactly this maintenance loop: a continuously re-scanned skills directory with every version pinned to a commit SHA, per-agent loadouts so agents only see what you choose, and alerts when upstream skills change. Join the waitlist at skillzbench.com to be first in line when it launches.

Making It Stick: Automating the Routine Without Bulk-Update Traps

A quarterly routine only works if it survives contact with real deadlines. The moment skill maintenance depends on someone remembering to do it, it stops happening — and the research shows exactly what slips through when it does. Orca Security demonstrated that bulk update commands refresh every installed skill at once with no per-skill granularity, no diff review, and no version pinning, and researchers there chained that blind spot into attack flows achieving actual code execution on end-user systems.

The first fix is to make updates deliberate. Pin each skill to a specific commit SHA rather than a mutable "latest," and treat any name collision from a different repository as hostile. The second fix is to shrink what each agent can see: per-agent loadouts mean an agent only receives the skills and tools explicitly toggled for it, which operationalizes the least-privilege principle Mondoo recommends when it advises treating skills as third-party code and building internal approved catalogs. A skill your agent cannot see is a skill that cannot hurt you.

Automation closes the remaining gap, because the threat is not static. The "Delayed Weaponization via Update" pattern — publish something benign, then swap in malicious code days or weeks later — is precisely why yesterday's clean verdict expires. And keyword-based scanning is trivially bypassed through simple obfuscation, so re-scans need intent-aware tooling, not pattern matchers that a construct like c${u}rl defeats.

A maintenance routine that sticks, in practice, looks like this:

• Pin every skill version to a commit SHA and require a human-reviewed diff before any update — never bulk-refresh.
• Give each agent its own loadout of explicitly approved skills and tools, so nothing is reachable by default.
• Automatically re-scan and re-benchmark when an upstream skill changes, and alert the owner instead of silently pulling the new version.
• Reconcile your inventory quarterly against what agents actually load, removing anything unrecognized.

This is exactly the workflow SkillzBench is building: a free directory where every version is pinned to a commit SHA, validated, license-checked, and security-scanned; a free MCP connector so agents only see the skills you toggle for them; and automated tracking of upstream changes with re-scanning, re-benchmarking, and notifications when something you rely on shifts. The platform is pre-launch, but the waitlist is live for teams who would rather have the routine handled than hand-rolled.

Skills, proven — that is the standard your agents should be held to, every quarter, without anyone having to remember.