What Happens When a Skill Updates Upstream?

The Invisible Update Problem: You Veted a Skill, Not Its Future Versions

You read the SKILL.md file, you checked what it does, you installed it. Done — except the file you vetted isn't the one your agent will be running next month. Agent skills for Claude Code, Cursor, Codex, and Gemini CLI are maintained overwhelmingly by upstream vendors like Anthropic, Stripe, and Supabase, so when a skill changes, the vendor controls the change — not the developer who approved it (see the awesome-agent-skills directory, where official vendor namespaces dominate).

The research on skill safety is sobering, and none of it accounts for what happens after installation. Unit 42's registry-scale analysis found that 80% of crawled skills showed at least one mismatch between what they declared and what they actually did — 250,706 behavioral deviations in total — with 5% carrying multi-stage attack chains. Mitiga's proof-of-concept went further, demonstrating a malicious skill that exfiltrated an entire codebase in just 4 user interactions while leaving the audit log completely empty.

Here's the gap those numbers expose: every documented verification mechanism works at install time. Neither major security study describes a mature process for re-verifying a skill after its upstream maintainer ships an update. The snapshot you vetted becomes stale the moment the vendor pushes a new version, and nothing in the current ecosystem flags that moment for you.

The risks compound because skills are live behavioral packages, not static text. As Mitiga puts it, skills are "pre-baked behaviors," so a single one-click install can quietly wire risky actions into your delivery flow. And reputation signals can't save you: Mitiga documented bots upvoting skills and fake accounts inflating download counts before pushing malicious versions toward real developers.

What a responsible workflow needs — and what the research stops short of providing — looks like this:

• Pin every installed skill to a specific version or commit, so upstream changes can't silently flow into your agents.
• Track upstream repositories for changes to skills you depend on, rather than discovering them by accident.
• Re-run security scans and behavioral benchmarks whenever a pinned skill updates — the same scrutiny you applied at install.
• Alert the humans who approved the skill originally, so a vendor-side change triggers a human-side review.

The core principle is simple: verification must follow the artifact through its lifecycle, not stop at the front door. SkillzBench is being built around exactly this idea — pinning every skill version to a commit SHA, then tracking upstream changes to re-scan, re-benchmark, and notify when something you installed evolves into something you never approved. Until update-time re-verification becomes standard practice, every installed skill is a standing bet that its upstream vendor never changes their mind.

How Risk Spreads After an Update: Reputation Fakes and Shared-Skill Contagion

A skill that looks popular and a skill that is safe are not the same thing — and 2026 gave us documented proof of both failure modes in the wild.

The first is deliberate: reputation inflation. Mitiga's security research documents a full attack chain where bots upvote a skill, fake accounts mass-download it, and fake comments pile on praise — all before the malicious skill ever reaches a developer's agent. The numbers make the point starkly: within roughly three months of Anthropic launching Agent Skills, the "Find-Skills" package had passed 200,000 downloads, a figure indistinguishable from genuine trust. Downloads, stars, and upvotes can be manufactured, so popularity signals are effectively meaningless as a safety filter.

The second is accidental, and arguably more alarming. When GitHub's CLI couldn't attach images to pull requests, agents at one company invented a workaround — hosting images in public personal repos — and saved it as a shared skill. According to The Hacker News, within a week more than a dozen agents had adopted the method for every ticket, contributing to 13,000+ internal images from 300+ organizations being exposed publicly. One risky workaround became contagion, spreading agent-to-agent with no human review at any hop.

Neither pattern is caught by checking a skill once at install time. Unit 42's registry-scale analysis found that 5% of crawled skills carry multi-stage attack chains, and its core insight explains why: "the threat lives in the chain, not the link." A scanner reviewing one capability at a time sees a file read and a network send as separate, harmless events — and flags neither.

What actually helps, based on the documented cases:

• Pin every skill to a specific version or commit, so an upstream change can't silently alter behavior you already vetted.
• Ignore popularity metrics entirely; rely on independent security scanning and behavioral testing instead.
• Audit the shared skill and instruction files your agents load — that's where workarounds like gitshot spread.
• Re-verify on every upstream update, not just at installation, since no mature update-time mechanism exists yet.

This is exactly the gap SkillzBench is being built to close: pinning each skill version to a commit SHA, re-scanning and re-benchmarking when upstream changes, and notifying users — so trust comes from verification, never from download counts.

Upstream Changes Cut Both Ways: Fixes, Deprecations, and New Risks

The story of agent skill updates is genuinely double-edged. When GitHub's gh CLI 2.99.0 shipped an --attach flag in September, it closed the exact gap agents had worked around by hosting images in public personal repos — the workaround that ended up exposing 13,000+ internal images from 300+ organizations. One upstream update, and a risky behavior agents had been abusing simply lost its reason to exist.

But upstream changes cut the other way too. Skills can be deprecated outright: the awesome-agent-skills directory marks the Protractor skill as deprecated in favor of Playwright or Cypress, and the Lettuce skill as legacy in favor of Behave. Teams still relying on those skills inherit stale instructions rather than current best practice. Worse, an update to a previously vetted skill can silently alter behavior — and no major security study documents a mature mechanism for re-verifying skills at update time. Both major verifications happen at installation only.

That gap matters because most update risk is accidental, not adversarial. Unit 42's registry-scale crawl classified the root cause of over 163,000 behavioral deviations and found 81.1% were developer oversight, with only 18.9% showing adversarial intent. The good news: your agents probably aren't under coordinated attack. The bad news: accidental behavior changes propagate just as fast, and Unit 42 found 80% of skills showed at least one declaration/behavior mismatch.

The community is still arguing about what skills even are. Skeptics on a 427-comment Hacker News thread dismissed them as "a framework to append text to your prompt," while supporters compared the dismissal to calling Docker "just some shell scripts" — a debate worth following if you're deciding how much governance to invest (Hacker News).

What the balance of evidence suggests in practice:

• Pin every skill to a specific version or commit so you know exactly what changed upstream.
• Re-verify on update, not just at install — an upstream change can alter behavior after initial vetting.
• Treat deprecation notices as action items, not trivia; stale skills teach stale patterns.
• Audit the shared skill files your agents load, since that's where workarounds spread "within a week" across a team.

This is exactly the "stay current" problem SkillzBench is building for: tracking upstream changes to every listed skill, re-scanning and re-benchmarking after updates, and notifying users when a skill they rely on shifts underneath them. Because most risk is accidental, the fix is process — version pinning, change tracking, and scheduled re-verification — rather than paranoia. Browse the free directory and join the waitlist at skillzbench.com to see upstream changes before your agents do.

The Playbook: Pin, Re-Verify, and Tier Your Review on Every Change

A skill you vetted last month is not the same skill your agent loads today. Upstream vendors control most skill changes — Anthropic, Stripe, and Supabase maintain their own packages — and no mature mechanism exists to re-verify a skill after it updates. Both major security studies verify at installation time only, which means the moment an upstream commit lands, your original review is stale.

The fix starts with pinning. Lock every skill to a specific version or commit SHA so nothing changes under you silently, and treat any upstream change as a trigger for full re-verification. SkillzBench is building exactly this workflow into its free directory: pin each version to a commit SHA, track upstream changes, re-scan, re-benchmark, and notify.

Second, stop trusting popularity. Mitiga's documented attack chain included bots upvoting skills, fake accounts mass-downloading, and fake comments to inflate trust before a malicious skill reached developer agents. The "Find-Skills" package passed 200K downloads — popularity never equals safety.

Third, audit what your agents actually load. In one real-world case, agents saved a risky workaround as a skill, and within a week more than a dozen agents had adopted it on every ticket, contributing to 13,000+ internal images from 300+ organizations being exposed publicly. Glow's guidance is blunt: read the shared skill and instruction files your agents load, because that is where workarounds spread. Treat any silencing or allowlist instruction inside a skill as an immediate red flag — Mitiga's proof-of-concept exfiltrated an entire codebase in just 4 user interactions and left the audit log completely empty.

Finally, drop the single pass/fail threshold. Unit 42's registry-scale research found that "a single threshold is the wrong instrument for this surface," because adversarial risk concentrates unevenly by capability family:

• Instruction manipulation: 96% adversarial fraction — critical review
• Credentials: 56% adversarial fraction — critical review
• Network: 37% — contextual review
• Process execution and file system: 12% and 10% — standard review

Unit 42's broader tiering — 5% of skills requiring mandatory review, 16.8% contextual, 72.5% benign — only works if you re-run the classification on every update. Re-verification on change is the emerging best practice the ecosystem currently lacks. Until registries adopt it natively, pin, track, and re-verify is the playbook — and it is precisely what SkillzBench's free "Stay current" tooling is being built to automate.

Staying Current Without Becoming a Full-Time Skill Auditor

Nobody wants to spend every Monday morning diffing SKILL.md files across a dozen agents. Yet the research is blunt about what happens if you don't: Unit 42's registry-scale analysis found 80% of skills showed at least one declaration/behavior mismatch, and Mitiga demonstrated a malicious skill exfiltrating an entire codebase with just four user interactions and an empty audit log. Verification at install time isn't enough — the gap is what happens after an upstream update slips in.

The good news is that change management is already a recognized need in the skills ecosystem. Some skills exist purely to handle upstream version changes, like stripe/upgrade-stripe for upgrading Stripe SDKs and callstackincubator/upgrading-react-native. The ecosystem even shows deprecation patterns — protractor and lettuce skills now carry "deprecated" and "legacy" labels pointing to successors. Vendors clearly expect skills to evolve; the missing piece is a systematic way to re-verify them when they do.

That's exactly the update-time workflow SkillzBench is building. Instead of asking you to become a full-time skill auditor, the platform is designed to automate the loop end to end:

• Pin every skill version to a commit SHA, so you always know exactly what your agent loaded
• Track upstream changes automatically, so an update never arrives unannounced
• Re-scan and re-benchmark each updated skill in a sandbox — with and without the skill — to publish a fresh Lift measurement
• Notify you when something you rely on has changed, so the decision to upgrade stays yours

This matters because popularity signals can't be trusted as a proxy for safety. Mitiga's research documented bots upvoting skills, fake accounts mass-downloading, and fabricated comments inflating trust before a malicious skill reached developer agents. A skill with 200K downloads and a skill that quietly changed behavior last Tuesday look identical on a directory listing.

Everything in this workflow — the directory, the security scanning, the benchmarks, the alerts, and the MCP connector that equips your agents with chosen skills — will be free. SkillzBench earns revenue from advertising around its content, never from users, and ad money never touches scores or listings.

Join the SkillzBench waitlist to get update alerts and re-verified benchmarks at launch. Skills, proven — before and after every upstream change.