Why Your AI Stack Needs a Loadout (Like a Game Character)

The 900,000-Skill Buffet Is a Trap

In under a year, the agent skills ecosystem has gone from a handful of GitHub repos to more than 900,000 indexed skills spread across eight major marketplaces. The default behavior this encourages — browse, install, repeat — feels productive. It's the trap.

The numbers behind that trap are uncomfortable, and no single study gives a universal rate. Snyk's ToxicSkills scan of 3,984 skills found 36.82% carried at least one security flaw, 13.4% critical. SmartScope's independent scan of SkillsMP found a 26.1% vulnerability rate — a figure that matches the ~26% published by SkillzBench's own research into skill security. And a 22,511-skill audit across skills.sh, ClawHub, GitHub, and Tessl averaged 6.3 issues per skill. Different samples, different methods — but every audit lands in the same danger zone.

Catalog size makes it worse, not better. The largest marketplace indexes 800,000+ skills scraped from GitHub and filtered by little more than "has 2 stars," which is why catalog size is the most misleading metric in this category. Anthropic's official catalog, by contrast, holds roughly 20 manually curated skills. Popularity signals don't rescue you either: Orca Security showed install counts can be inflated with a single unauthenticated HTTP request, and Snyk's own scanner marked a deliberately malicious test skill "CLEAN. 0 findings" because pattern-matching misses obfuscation like c${u}rl instead of curl.

Bulk-installing also fails on performance and cost, not just security. NVIDIA's SkillEvaluator found the right skills lift agent scores by +39–41 points — but per-product lift ranged from +2 to +46, and a skill that loads when it isn't relevant actively reduces performance. Token overhead compounds the damage: one practitioner measured a full "everything" install consuming 50,000 tokens before a single character was typed, then cut it to 13,000 through curation — a 44% reduction.

The fix is curation, and it's smaller than you think. The practitioner filter that cuts 900,000 options down to about 20:

• Trusted publisher with a verifiable identity
• Actively maintained, not an abandoned experiment
• Solves a real problem you actually have
• SKILL.md read before install, every time

This is exactly the gap SkillzBench is being built to close: continuously discovering and deduping skills from GitHub, pinning every version to a commit SHA, security-scanning and quality-judging each one, and benchmarking every skill in a sandbox — with and without the skill — so the Lift in percentage points is always visible before you equip it. Per-agent loadouts through a single MCP server mean your agent only sees what you deliberately chose.

Skills, proven. Join the SkillzBench waitlist to get benchmarked Lift scores and per-agent loadouts the moment we launch.

Published research found ~26% of agent skills carry at least one vulnerability — SkillzBench benchmarks every skill so you equip only the ones that earn their slot.

One Approved Skill Can Run Ransomware: Why Trust Signals Lie

In a controlled demo, Cato CTRL took Anthropic's own GIF Creator Skill, made a minor modification, and used it to silently deploy MedusaLocker ransomware — all under a single user approval, with no secondary prompts even in strict mode. The victim's only "mistake" was installing a skill that looked legitimate. With the average ransomware incident costing $5.08 million per Cato CTRL's research, one careless install can become an enterprise catastrophe.

The scariest part is what Cato CTRL observed: "The key issue isn't that the Skill runs code. It's that visibility stops at what's shown." You approve a GIF tool; the agent executes something else entirely.

And the trust signals you'd naturally rely on? They lie.

Orca Security demonstrated that install counts can be inflated with a single unauthenticated HTTP request — so marketplace popularity is trivially fake. Their research also surfaced three more systemic failures:

• Security scans run only at creation and at a popularity threshold, leaving exploitation windows of days to weeks.
• Installing a skill with the same name silently overwrites whatever you already had — a perfect bait-and-switch vector.
• Bulk updates pull upstream changes with no diff review and no version pinning, so a trusted skill can turn hostile on its next refresh.

Automated scanners don't save you either. In testing, Snyk's Skill Defender marked a deliberately malicious skill "CLEAN. 0 findings" — because it pattern-matches known threats and can't catch simple obfuscation like c${u}rl standing in for curl.

The deeper problem is that SKILL.md is an entirely new attack surface. As one security writeup put it, "a malicious npm package needs to exploit a code vulnerability. A malicious SKILL.md just needs to write a convincing English sentence." The agent reads instructions and follows them — no exploit required. And because the risk travels with the file, not the platform, the same poisoned skill works identically in Claude Code, Codex, Cursor, or Gemini CLI.

This is why SkillzBench treats every skill as untrusted by default: pinning each version to a commit SHA, security-scanning continuously rather than once, and re-scanning whenever upstream changes — because a skill approved in January may not be the skill you're running in March.

Until verification tooling matures, your best defense is skepticism. Read the SKILL.md before installing, pin versions explicitly, treat name collisions as hostile, and never bulk-update blind. Anthropic's own position is blunt: "It is the user's responsibility to only use and execute trusted Skills." Treat every skill as untrusted code — because that's exactly what it is.

The Loadout Thesis: Right Skills Lift +41 Points, Wrong Ones Hurt

The strongest argument for loadouts isn't safety — it's winning. NVIDIA's SkillEvaluator, which tested 300+ verified skills across 30+ products, found that the right skills transform agent performance: an average lift of +41 points on Correctness and +39 on Effectiveness, moving baseline scores of 39–46/100 into the 78–87 range, according to NVIDIA's benchmarking research.

But the averages hide a spread that matters enormously. Per-product lift ranged from +2 to +46 points, and NVIDIA's central finding cuts against bulk installation: "a skill that loads when it is not relevant can reduce agent performance." Every irrelevant skill you equip isn't neutral baggage — it actively competes for the agent's attention and degrades results. As NVIDIA puts it, agents are only as effective as the context they receive.

The token-budget mechanic reinforces this. One practitioner documented that a full everything-claude-code install consumed 50,000 tokens before typing a single character; manually curating down to a lean set cut that to 13,000 tokens — a 44% reduction — per first-party measurements. Token impact varies wildly per skill, too: NVIDIA measured one skill cutting tokens by 76.9% while another increased consumption by 120.3%.

The game-mechanics parallel is exact:

• Skills have a cost — tokens are your carry-weight budget, and every equipped skill spends it every turn.
• Skills have variable value — lift ranges from +2 to +46 points, so slot choice is the whole game.
• Wrong equipment hurts — irrelevant skills don't just waste budget, they reduce performance outright.

One more finding shapes how loadouts should be built: the agent harness barely mattered. Claude Code and Codex differed by only ~5 points in NVIDIA's testing, while domain and task drove the biggest differences. That means loadouts should be task-specific, not agent-specific — your "code-review agent" and your "data-pipeline agent" need different kits, even on the same platform.

This is precisely why SkillzBench is building per-agent loadouts with published Lift scores: the question "which skills actually help this task?" needs a measured answer, not a guess. Equip the skills that earn their slot, and leave the rest in the inventory.

Skills, proven. Join the SkillzBench waitlist to get benchmarked Lift scores and per-agent loadouts the moment we launch.

Building Your Loadout: A Curation Checklist That Treats Skills as Code

Every skill you install is code, whether or not it contains a script. A malicious SKILL.md just needs to write a convincing English sentence, so the curation checklist has to treat skills as executables from day one.

Start with the practitioner's filter. A well-known practitioner breakdown argues that a trusted publisher, active maintenance, a real problem it actually solves, and a full read of the SKILL.md before install cuts roughly 900,000 indexed skills down to about 20 — and 20 is plenty. Excluding mega-collections and unverifiable authors is part of the filter, not an optional extra.

Then layer on security-team hygiene:

• Run agents in a sandbox and monitor file writes, subprocesses, and outbound connections, per Cato CTRL's ransomware analysis.
• Pin every skill version to a commit SHA and avoid blind bulk updates, which Orca Security showed can silently overwrite existing skills with no diff.
• Treat name collisions as hostile — a same-name install can replace a trusted skill without warning.
• Never rely on popularity. Orca demonstrated install counts can be inflated with a single unauthenticated HTTP request, and a scanner marked a deliberately malicious test skill "CLEAN. 0 findings."

This is exactly the gap SkillzBench is building against. Its five pillars — Discover, Verify, Benchmark, Equip, Stay current — map one-to-one onto the documented failures: Verify pins every version to a commit SHA with security scanning and license checks, answering Orca's call for package-manager rigor. Benchmark tests each skill in a sandbox with the same model, with and without the skill, publishing the Lift in percentage points — mirroring NVIDIA's methodology, which found average lifts of +41 points for Correctness but per-product results ranging from +2 to +46, with irrelevant skills actively reducing performance. Equip delivers one MCP server with per-agent loadouts, so each agent only sees the skills and tools you toggled for it — the delivery model analysts expect to become the default within 12 months.

The checklist works today by hand. SkillzBench is building it so it runs automatically — and the waitlist is open if you want loadouts and Lift scores the moment they ship.

From Installs to Loadouts: Where the Ecosystem Is Heading

The install-skill-and-hope era is ending. Agensi.io's market analysis predicts that within 12 months, skills will be delivered through MCP servers rather than downloads — agents connect once and load skills live, because download-based distribution becomes operationally heavy past 20–30 installed skills. That's exactly the loadout model: per-agent skill sets, toggled on and off, with the agent only seeing what was deliberately chosen.

The ecosystem itself is due for a shakeout. Eight major marketplaces have already emerged in under a year, and the prediction is consolidation down to three or four dominant platforms, with security scanning becoming a selling point rather than an afterthought. After Snyk's ToxicSkills research found that 36.82% of scanned skills carried at least one security flaw — and that a deliberately malicious test skill was marked "CLEAN. 0 findings" by a pattern-matching scanner — nobody can claim trust signals alone are enough.

That last point matters for anyone running agents in an organization. Enterprise security teams increasingly evaluate agents on three criteria: controllable, observable, and auditable. A stack of 900,000 loosely governed instruction files fails all three. Anthropic's own position doesn't help: "It is the user's responsibility to only use and execute trusted Skills." If the platform vendors put the burden on you, independent verification is the only way to carry it.

That's the gap SkillzBench is being built to close. Its Equip pillar pairs one MCP server with per-agent loadouts, while Verify pins every skill version to a commit SHA, runs security scans and license checks, and Benchmark tests each skill in a sandbox — with and without it — publishing the Lift in percentage points, mirroring NVIDIA's SkillEvaluator methodology, which found per-product lift ranging from +2 to +46 points. Money never touches the scores, and Lift is always shown separately from ratings.

The trajectory is clear:

• Delivery shifts from downloads to MCP-based, per-agent skill access within the year.
• Marketplaces consolidate from eight down to three or four serious players.
• Continuous security scanning and version pinning become baseline expectations, post-ToxicSkills.
• Governance — controllable, observable, auditable agents — becomes the enterprise gate.

Skills, proven. Join the SkillzBench waitlist to get benchmarked Lift scores and per-agent loadouts the moment we launch. Published research found ~26% of agent skills carry at least one vulnerability — SkillzBench benchmarks every skill so you equip only the ones that earn their slot.