The Skills Gold Rush: Quality Problems at Scale

The Gold Rush Outpaced the Safety Rails

The gold rush is real — and it's moving faster than anyone can vet what's being dug up. The AI agents market is projected to grow from $11.55 billion in 2026 to $294.66 billion by 2035, a staggering 43.57% CAGR, according to market research. Meanwhile, non-developer adoption of coding agents grew 137x for individuals and 189x for organizations in under a year, per OpenAI's own usage data.

Into that vacuum, agent skills — SKILL.md instruction packages that extend what agents like Claude Code, Codex, and Cursor can do — have proliferated across marketplaces with essentially no gatekeeping. The first large-scale empirical study of the ecosystem makes the consequences concrete. Researchers collected 42,447 skills from two marketplaces and analyzed 31,132 unique ones. The result: 26.1% of analyzed skills carry at least one vulnerability, with data exfiltration patterns in 13.3% and privilege escalation in 11.8%, according to the study. Most troubling, 5.2% exhibit high-severity patterns that strongly suggest malicious intent.

The structural problem isn't a few bad actors — it's the marketplace model itself. These platforms aggregate third-party skills from public repositories with no mandatory vetting, which the researchers compare directly to early browser extension stores, where security considerations were secondary to functionality. And the "consent gap" makes it worse: once you approve a skill, it gains persistent permissions without further prompts. Anthropic's own documentation acknowledges skills can enable agents to exfiltrate data and take unintended actions, recommending users install only from trusted sources and thoroughly audit untrusted ones.

Real attacks have already landed. In December 2025, researchers demonstrated a benign-looking "GIF Creator" skill silently downloading and executing MedusaLocker ransomware. A separate campaign showed state-sponsored actors weaponizing agents through malicious MCP servers, with 80–90% of tactical operations running autonomously.

Even Google, with resources most teams can only dream of, names quality control as the central scaling challenge. "A skill is a living product, not a one-off document," Google's team writes, warning that without automated governance, an open-source skills repository quickly becomes chaotic. Google runs continuous regression evaluations and uplift testing — comparing agent performance with and without each skill — as internal countermeasures.

That's exactly the gap this ecosystem needs filled for everyone else. SkillzBench, an independent testing lab from AIQ Labs, is building precisely this layer: continuous discovery of skills from GitHub, version-pinned verification with security scanning, and sandboxed benchmarking that measures each skill's Lift — with and without it — under identical conditions. The trust infrastructure hasn't caught up to the gold rush, and closing that gap is the market's next bottleneck.

Until then, every install is an unverified bet.

The Consent Gap: Why One Click Lasts Forever

The most dangerous thing about an agent skill isn't what it does when you install it — it's what it can do forever after. Researchers studying more than 31,000 agent skills identified a structural flaw they call the consent gap: once a user approves a skill, it gains persistent read, write, and network permissions without ever prompting again (the first large-scale study of agent skill vulnerabilities).

That one-click approval model turns every install into a permanent grant. The study's authors conclude there is "an urgent need for capability-based permission systems and mandatory security vetting" before this attack vector is exploited further. Until that happens, the trust problem is architectural — not just a matter of filtering out bad actors.

The consequences are no longer hypothetical:

• In December 2025, Cato CTRL researchers demonstrated a benign-looking "GIF Creator" skill silently downloading and executing MedusaLocker ransomware.
• The GTG-1002 campaign showed state-sponsored actors weaponizing Claude Code with malicious MCP servers, with 80–90% of tactical operations running autonomously.
• 5.2% of analyzed skills exhibit high-severity patterns the researchers say strongly suggest malicious intent.

The math gets worse when skills carry executable code. Script-bearing skills are 2.12x more likely to contain vulnerabilities than instruction-only packages, a statistically significant finding (p<0.001) from the same research. Data exfiltration appears in 13.3% of analyzed skills and privilege escalation in 11.8% — and marketplaces currently aggregate third-party skills with no mandatory vetting, a dynamic the authors compare to early browser extension stores where security took a back seat to functionality.

Meanwhile, the population of people installing these skills is exploding beyond the experts who could audit them. Non-developer agent adoption grew 137x for individuals and 189x for organizations in under a year, according to OpenAI's research on agent usage — meaning millions of users are approving permissions they have no practical way to inspect.

This is why the problem can't be solved by warnings alone. Anthropic's own documentation acknowledges skills "can introduce vulnerabilities" and recommends users install only from trusted sources — but "trusted" is exactly the signal the ecosystem lacks. It's the gap SkillzBench is building to close: pinning every skill version to a commit SHA, running security scans and quality judgments, and designing per-agent loadouts where the agent only sees the skills a user deliberately chose. At launch, SkillzBench will list script-bearing skills link-out only, reflecting the elevated risk the data shows.

A permission you grant once shouldn't be a permission you can never reconsider. Fixing that requires infrastructure, not vigilance — and the ecosystem is only beginning to build it.

Skills, proven. Join the SkillzBench waitlist to get verified, benchmarked skill intelligence at launch.

What Google's Playbook Reveals About Quality at Scale

When Google — a company with essentially unlimited engineering resources — publishes a blog post admitting that quality control is its hardest scaling problem, the rest of the ecosystem should pay attention. Their candid account of building, testing, and scaling agent skills reads less like a victory lap and more like a warning label.

Google's Remigiusz Samborski puts it plainly: "a skill is a living product, not a one-off document." In their engineering write-up, the team states that "popularity brings a major challenge: quality control," because "a poorly written skill with vague instructions, broken links, or missing edge cases degrades the entire agent experience."

Their countermeasures reveal what serious quality infrastructure actually looks like:

• Uplift testing: a 2x2 matrix comparing agent performance with and without each skill, on both accuracy and efficiency
• On-submit evaluations plus CI/CD linters and link checkers before any skill ships
• Weekly full-library regression evals, because "a skill that works today might break tomorrow if an underlying API, model, or agent harness changes"

That last point deserves emphasis. Google re-tests its entire skill library every single week — not because its skills are bad, but because the ground beneath them keeps shifting. Skills interact with models, APIs, and agent harnesses that evolve independently. Verification is not a gate you pass once; it is a continuous process.

Here is the uncomfortable implication. If Google needs this machinery internally to keep 15,000+ starred skills trustworthy, what about everyone else? The broader ecosystem — marketplaces aggregating third-party skills "with no mandatory vetting," where researchers found 26.1% of 31,132 analyzed skills contain at least one vulnerability — has no equivalent infrastructure at all.

And the timing matters. According to the Stanford HAI AI Index, the score gap between the top and 10th-ranked models fell from 11.9% to 5.4% in a single year. As models converge, raw intelligence stops being the differentiator; the quality of the surrounding skill layer becomes the differentiator. An unverified layer on top of commoditized models is precisely where value leaks out.

This is the gap SkillzBench was designed to close: an independent, democratized version of the uplift testing, security scanning, and continuous re-evaluation that Google built for itself — available to every team, not just the ones with a platform engineering division.

Verification Infrastructure Is the Missing Layer

If 26.1% of skills contain at least one vulnerability — and marketplaces aggregate them with no mandatory vetting — the ecosystem doesn't need more skills. It needs a verification layer that tells you what's safe and what actually works.

That's the gap SkillzBench is being built to close. The approach starts where the research points: pinning every skill version to a commit SHA, so what you evaluate is exactly what you install. From there, each pinned version gets license checks, security scanning, and quality judging before it ever reaches a benchmark.

The benchmarking methodology mirrors what Google built internally for its own skills program. Google's team compares agent performance with and without each skill across accuracy and efficiency, then runs weekly regression evaluations across the entire library — because, as their engineers put it, "a skill that works today might break tomorrow". SkillzBench is building the same discipline for everyone: sandbox testing with the same model, run twice — once with the skill, once without — and the difference published as a Lift score in percentage points.

Three design principles keep that number honest:

• Lift is always shown separately from trust scores — a skill can be popular and harmful, or mediocre and safe, and you deserve to see both dimensions.
• Money never touches scores — no paid placement, no sponsor-influenced ratings.
• Instruction-only at launch — script-bearing skills are listed link-out only, reflecting the finding that skills bundling executable scripts are 2.12x more likely to contain vulnerabilities (per the first large-scale security study of agent skills).

Verification can't be one-and-done, either. Skills are living products whose upstream repositories change without notice, so SkillzBench is building continuous tracking: when an author pushes a change, the skill gets re-scanned, re-benchmarked, and users get notified. And because the research identified a "consent gap" — approved skills gaining persistent permissions with no further prompts — the Equip layer gives users per-agent loadouts, where each agent only sees the skills and tools explicitly chosen for it.

None of this is live yet. SkillzBench is pre-launch, and the waitlist is open for teams that want verified, benchmarked skill intelligence from day one.

Skills, proven — join the waitlist at skillzbench.com.

How to Evaluate Skills Before You Install Them

Installing a skill takes thirty seconds. Auditing it properly takes far longer — and the numbers say most people skip it entirely, which is exactly what attackers and negligent authors are counting on.

Start with the highest-signal filter: script bundles. The first large-scale security study of agent skills analyzed 31,132 unique skills and found that skills bundling executable scripts are 2.12x more likely to contain vulnerabilities than instruction-only packages. If a skill carries scripts, either drop it or read every line before you approve it. Remember the consent gap: once you approve a skill, it holds persistent permissions without further prompts.

Second, hunt for vague instructions. Google — which runs one of the largest skills programs anywhere — warns that "a poorly written skill with vague instructions, broken links, or missing edge cases degrades the entire agent experience." Open the SKILL.md and ask: are the steps concrete? Do referenced links resolve? Does the author handle edge cases, or hand-wave them? A skill that only works on the happy path is a liability dressed as a shortcut.

Third, apply Anthropic's own guidance: install only from trusted sources, and thoroughly audit anything untrusted. The study found 26.1% of skills carry at least one vulnerability, with data exfiltration at 13.3% and privilege escalation at 11.8% — and marketplaces currently aggregate third-party skills with no mandatory vetting. Your default posture should be skepticism.

Fourth, check whether the skill is pinned to a specific commit. A skill without version pinning can change under you at any moment — and per Google, "a skill that works today might break tomorrow if an underlying API, model, or agent harness changes." Pinning gives you a fixed reference point; no pinning means no accountability.

Finally, treat every installed skill as a living product. Google re-runs full-library regression evaluations weekly and tests each skill's uplift with and without it. Re-evaluate your skills after model updates, API changes, or harness upgrades — a skill that helped last month may silently hurt today.

Here's your quick audit routine:

• Reject or fully read any skill bundling executable scripts — they carry 2.12x the vulnerability risk
• Open the SKILL.md and check for vague instructions, broken links, and missing edge cases
• Install instruction-only skills from trusted sources; audit everything else line by line
• Prefer skills pinned to a specific commit SHA over unversioned installs
• Re-evaluate skills after every model, API, or agent-harness change

These steps work, but they're manual — and the ecosystem is scaling past what individuals can audit. Consider the gap: 93% of IT executives express interest in agentic AI while only 37% actually use it. That's not a demand problem. It's a trust problem.

That's the bottleneck SkillzBench is building against: an independent testing lab that pins every skill version to a commit, security-scans and quality-judges instruction-only skills, and benchmarks each one in a sandbox — with and without the skill — publishing the Lift in percentage points, shown separately from ratings, with money never touching scores. Skills, proven. Join the SkillzBench waitlist at skillzbench.com to get verified, benchmarked skill intelligence at launch.