The Case for Pinned Versions in Agent Skills

The Moving Target: Why Agent Skills Can't Be Trusted at Head

The skill you reviewed yesterday may not be the skill running in your agent today. That's not a hypothetical — it's the defining structural weakness of the agent-skills ecosystem, and it's getting worse as the ecosystem grows.

The growth itself is staggering. According to Snyk's research, daily skill submissions jumped from under 50 in mid-January 2026 to over 500 by early February — a 10x increase in a matter of weeks. And the publishing barrier is effectively zero: a SKILL.md file and a week-old GitHub account, with no code signing and no mandatory security review.

The numbers behind that flood are sobering. A large-scale academic analysis of 31,132 skills found that 26.1% contain at least one vulnerability, with 13.3% exhibiting data exfiltration patterns and 11.8% privilege escalation. Snyk's separate scan of 3,984 skills found 13.4% carrying critical security issues — and 2.9% use unverifiable runtime dependencies like curl | bash, meaning a skill can appear benign during review and change its behavior remotely afterward.

Worse, compromise doesn't require a malicious author. AIR's "SkillJacking" research documented 925 skills hijacked from their original maintainers, reaching 134,000 agents before detection. A benign, well-reviewed skill can turn hostile overnight through account takeover or repository hijacking — and unlike traditional packages, skills load and execute dynamically based on task context, shrinking the window for human review.

The core problems compound each other:

• No gate at entry — community marketplaces aggregate third-party skills without mandatory security review.
• No stability after entry — runtime dependencies and upstream changes let behavior shift post-installation.
• No verification at install — the Plugin4Shell disclosure showed agents request pinned SHAs but never confirm the checkout matches.

Unit 42 puts it plainly: the ecosystem stands where browser extensions were a decade ago, with extensibility having outpaced the supply-chain audit primitives that should gate it. Their scan of 49,943 registry skills found 80% with declared-vs-actual behavioral mismatches — though most trace to developer sloppiness, 5% carry multi-stage attack chains.

This is the moving target problem: a review performed once is a review that expires. SkillzBench's approach — pinning every version to a commit SHA, then re-scanning and re-benchmarking as upstream changes — is built on exactly this premise. When the artifact itself can change under you, the only trustworthy unit is a pinned, verified, re-checked version.

What Pinning Actually Buys You — and Where It Stops

Pinning a skill to a commit SHA gives you something the agent-skills ecosystem badly lacks: a frozen, reviewable artifact. Once you review a specific version, that exact version is what installs — not whatever the maintainer pushes next week. That matters because hijacking is not hypothetical: AIR's "SkillJacking" research found 925 skills in active use had been taken over from their original maintainers, reaching 134,000 agents.

Pinning also defends against subtler post-review swaps. Snyk found that 2.9% of ClawHub skills use unverifiable runtime dependencies like curl | bash — patterns that let behavior change after review. A pinned version can't be silently rewritten.

But the research's sharpest finding is where pinning stops working. The Plugin4Shell disclosure showed that all four major coding agents requested pinned SHAs but never verified the checkout matched. Attackers could exploit Git reference ambiguity — say, a branch named like the pinned hash — to achieve zero-click RCE with full agent privileges. Because background auto-update is the default in Claude Code and Codex, a marketplace SHA bump reaches already-installed plugins with no user action at all.

The vendor response was uneven:

• Anthropic patched Claude Code in v2.1.179
• OpenAI patched Codex in v0.146.0
• Microsoft's Copilot was unpatched as of publication, with the status disputed between AIR and GitHub
• Google deprecated Gemini CLI rather than patching it

Even where pinning works, it only pays off with review discipline behind it. Security researcher Mark Curphey argues that pinning without reviewing updates is "security theatre": if you don't manually review the code when you update a dependency, "you might as well auto-update everything." His estimate — explicitly a guess — is that roughly 99% of teams skip that review.

The honest conclusion is that pinning is the necessary first step, not the whole answer. It creates the stable artifact that verification and re-review require — confirming the checkout matches the pin, and re-scanning every update. That pin → verify → re-review loop is exactly what SkillzBench is building into its Verify and Stay Current pillars, because a pin you never check is just a promise nobody kept.

Pin → Verify → Re-Review: The Three-Part Discipline

Pinning a skill version is like locking your front door while the burglar is already inside. Plugin4Shell proved the point: all four major AI coding agents requested a pinned commit SHA during installation, yet none verified the checkout actually matched it — enabling zero-click remote code execution with full agent privileges.

That's why pinning only works as the first step of a three-part discipline. The research points toward a pattern none of the sources names explicitly, but all of them imply: pin the version, verify the artifact matches the pin, and re-review every update. We present this as an interpretation — the underlying studies advocate vetting and verification, not pinning alone.

The verification step is where most of the ecosystem currently fails. Unit 42's scan of 49,943 registry skills found that 80% showed declared-vs-actual behavioral mismatches — the skill you think you installed is often not the skill you got. That's why Unit 42 recommends behavioral-integrity checks before installation, not after. The arXiv study reaches a similar conclusion, calling for mandatory security vetting before this attack vector is further exploited.

Re-review closes the loop. Mark Curphey argues that pinning without reviewing updates is "security theatre" — if you don't manually review each update, you might as well auto-update everything. He estimates roughly 99% of teams skip that review, though he labels the figure a guess. The risk is concrete: Snyk found that 2.9% of ClawHub skills use unverifiable runtime dependencies like curl | bash, letting attackers change behavior after review.

The discipline, in full:

• Pin — lock each skill to a specific commit SHA, creating a stable, reviewable artifact.
• Verify — confirm the installed artifact actually matches the pin, and run behavioral-integrity and security checks before installation.
• Re-review — when upstream changes, re-scan and re-review the new version before it reaches your agents.

This is exactly the workflow SkillzBench is being built to operationalize: pinning every version to a commit SHA, validating and security-scanning it, then tracking upstream changes to re-scan and re-benchmark each new release. Pinning gives you something stable to trust; verification and re-review are what make that trust earned rather than assumed.

Operationalizing the Discipline: How SkillzBench Is Building It

Pin, verify, re-review — that discipline is easy to prescribe and hard to run. Someone has to actually do the verifying, re-check every upstream change, and keep the review alive long after installation day. That operational burden is exactly why ~99% of teams, by one practitioner's estimate, skip code review when updating dependencies — and why the discipline needs to be built into infrastructure rather than left to individual diligence.

SkillzBench, an independent testing lab for AI agent skills, is building that infrastructure around two pillars that map directly onto the pin→verify→re-review loop. Under its Verify pillar, every skill version gets pinned to a specific commit SHA, then validated, license-checked, security-scanned, and quality-judged. The pin creates the stable, reviewable artifact the research demands; the scan and judgment are the review that pinning without review reduces to security theatre.

The Stay Current pillar closes the loop. Upstream changes trigger a re-scan and re-benchmark of the new version, with notifications when anything shifts — so a hijacked maintainer pushing a payload can't swap behavior silently under an unchanged pin. This addresses the exact gap Plugin4Shell exposed, where all four major coding agents requested pinned commit SHAs but never verified the checkout actually matched.

The launch scope reflects the data too. SkillzBench is starting with instruction-only skills — SKILL.md packages plus text references — and listing script-bearing skills link-out only. That's not caution for its own sake: a large-scale study of 31,132 skills found executable-code skills are 2.12× more likely to contain vulnerabilities than instruction-only ones. Instruction-level threats are also the most adversarial surface — 96% of instruction manipulation deviations traced to intent rather than oversight, per Unit 42's scan of nearly 50,000 skills.

Two design choices matter for trust:

• Lift — the measured performance gain from using a skill — is always shown separately from the trust score, so "does it help" and "is it safe" never blur into one number.
• Money never touches either score. Benchmark results and trust ratings aren't for sale, which matters in an ecosystem where marketplace incentives have historically favored volume over vetting.

This is the synthesis the research points toward but no single source states outright: a pinned, verified, re-reviewed skill version is the only installable unit you can actually reason about. SkillzBench is building the machinery to make that unit the default — and the waitlist is open for teams who want proven skills rather than plausible ones.

Your Practical Pinned-Skills Checklist Today

Pin first, verify second, review third — miss any step and the whole exercise collapses. Plugin4Shell proved that in September 2026: all four major coding agents requested a pinned commit SHA but never checked the checkout actually matched, enabling zero-click remote code execution. Here's the checklist to run today.

• Pin every skill to a commit SHA — and verify the checkout matches. Plugin4Shell showed attackers could exploit Git reference ambiguity, so confirm your working tree HEAD equals the pinned hash after every install. Claude Code (v2.1.179) and Codex (v0.146.0) shipped patches; check your agent version.
• Audit installed skills for declared-vs-actual behavior gaps. A Unit 42 scan of nearly 50,000 skills found 80% had at least one behavioral mismatch. Inventory what you've installed and compare documented behavior against what the files actually do.
• Treat every upstream bump as a new review event. Mark Curphey's argument is blunt: pinning without manually reviewing updates is "security theatre." Auto-update means a hijacked maintainer's swap reaches you with no user action — 925 hijacked skills reached 134,000 agents.
• Prefer instruction-only skills where possible. The data backs this: a large-scale skills study found script-bearing skills are 2.12× more likely to contain vulnerabilities than instruction-only ones.

Why go through this effort? Because the ecosystem is under active attack, not just theoretically fragile. A Snyk scan found 13.4% of skills carried at least one critical flaw, and 2.9% used unverifiable runtime dependencies like curl | bash — meaning behavior can change after you review. Pinning a reviewed version is the only defense against that post-review swap.

This discipline — pin, verify, re-scan on every change — is exactly what SkillzBench is building across its Verify and Stay Current pillars: pinning each version to a commit SHA, security-scanning it, and re-benchmarking when upstream changes. If you'd rather not run this checklist by hand across every agent and skill, join the SkillzBench waitlist to get notified when continuous pinning, verification, and benchmarking of skill versions goes live.

Skills, proven.