Who Verifies the Verifiers? Trust in AI Directories

The Trust Gap: AI Skill Directories Are Unvetted Aggregators

Imagine an app store where anyone can publish anything, no one reviews the code, and the fine print says "verify it yourself." That is effectively how today's AI skill directories operate.

Academic research is blunt about it: community registries aggregate third-party skills without mandatory security review, typically with no centralized governance or vetting. One large-scale study found registries indexed more than 98,000 skills within three months of launch — low friction, limited gatekeeping, and security review that rarely happens. Unlike npm's trusted publishing or PyPI's verified publishers, skill registries provide minimal provenance verification.

The numbers quantify what that means for anyone browsing a directory. A systematic analysis of 31,132 skills found that 26.1% contain at least one vulnerability, spanning 14 distinct patterns — including data exfiltration (13.3% prevalence) and privilege escalation (11.8%). Notably, skills bundling executable scripts are 2.12× more likely to carry vulnerabilities than instruction-only packages.

Worse, the vendors who host these ecosystems explicitly hand the problem to you. Anthropic's own documentation warns that skills "can introduce vulnerabilities" and advises users to "install only from trusted sources" and "thoroughly audit untrusted skills." Its security docs state plainly that Anthropic does not manage or audit the servers and skills connected to its agents. In practice, the answer to "who verifies?" is: you — the person least equipped to do it.

And the trust signals directories display can't fill the gap:

• Install counts can be inflated through unauthenticated telemetry endpoints, as Orca Security demonstrated, while non-deterministic scanning leaves malicious modifications undetected for days or weeks as clean audit badges keep displaying.
• Stars and upvotes are trivially gameable — Mitiga researchers warn adversaries can manufacture bot upvotes, fake downloads, and planted comments.
• Skills are not cryptographically signed and rarely properly vetted; anyone with a GitHub account can publish one, and agents will happily ingest and execute whatever's inside.

The verification burden has been quietly shifted onto end users — business owners, managers, and developers alike — while the ecosystem scales faster than any review process can follow. This is exactly the gap SkillzBench is being built to close: an independent directory where every skill is version-pinned, security-scanned, and behaviorally benchmarked before you ever equip it, rather than relying on popularity metrics that anyone can fake.

Why the Numbers Lie: Stars, Installs, and Badges Can't Be Trusted

Every directory shows them: star counts, install totals, "security verified" badges. They look like proof. In practice, they're the easiest thing on the page to fake.

Orca Security demonstrated exactly how fragile these signals are, identifying four attack primitives in a prominent AI skills marketplace — including install count inflation through unauthenticated telemetry endpoints. Worse, the marketplace's security scanning was non-deterministic, leaving exploitation windows of "days or weeks" in which malicious modifications went undetected while the skill's clean audit badge kept displaying.

Mitiga's guidance is blunt: "Don't trust skills based on their 'Stars' or 'Upvotes." Adversaries can manufacture credibility with bot upvotes, fake download accounts, and coordinated comments. A popular-looking skill is simply a skill that was made to look popular.

Even genuine audits have a blind spot. HiddenLayer found that code-focused reviews miss skills that are technically clean but behaviourally compromised — an affiliate link quietly inserted into every recommendation, for instance, or harmful instructions written in plain natural language rather than code.

The real-world results speak for themselves:

• A single actor was responsible for 54.1% of confirmed malicious skills, using templated brand impersonation to borrow trust from names users already recognize.
• A benign-appearing "GIF Creator" skill, demonstrated by Cato CTRL in December 2025, silently downloaded and executed MedusaLocker ransomware.
• Mitiga's proof-of-concept exfiltration attack required only four user interactions and produced completely empty audit logs.

The deeper problem is structural: static scanning — the verification layer most directories rely on — cannot distinguish malice from benign risk. One large-scale behavioral study of 98,380 skills found that while static tools flag thousands of skills, behavioral verification confirmed 157 as genuinely malicious, achieving 99.6% precision — a 150× improvement over static-only approaches.

This is why SkillzBench is being built around actually running skills, not just reading them: every skill gets tested in a sandbox with the same model, with and without the skill, and the resulting Lift in percentage points is always published separately from any rating or popularity metric. Popularity tells you what's loud. Testing tells you what's true.

What Real Verification Looks Like: Provenance, Scanning, and Behavioral Testing

If a scanner flags 26% of skills as risky but only 157 of 98,380 are actually malicious, what does "verified" even mean? That gap between static warnings and confirmed behavior is exactly where today's trust problem lives — and it's where real verification has to start.

The research is blunt about what doesn't work. Static scanning — the de facto verification layer on most directories — cannot distinguish intentional abuse from benign functionality, according to large-scale behavioral analysis. It flagged 26.1% of skills as potentially risky, yet behavioral verification confirmed just 157 malicious skills out of 98,380, with 99.6% precision — a 150× improvement over static-only approaches. Code-focused audits also miss skills that are "technically clean but behaviourally compromised," where the harm hides in natural-language instructions rather than malicious code.

The academic consensus calls for a specific verification stack. One study urges "mandatory security vetting" before publication; HiddenLayer researchers recommend app-store-style review plus cryptographic signing and verified provenance, mirroring modern package managers. In practice, that translates to a few concrete mechanisms:

• Version pinning — locking every skill to an exact commit SHA so what you reviewed is what you run, and upstream changes can't silently swap in new behavior.
• Continuous security and license scanning, re-run whenever upstream code changes rather than only at first publication.
• Behavioral testing — actually running the skill in a sandbox and comparing what the agent does with and without it.
• Transparent scoring, where the methodology itself is published and auditable.

That last point matters more than it sounds. Orca Security researchers showed install counts can be inflated through unauthenticated telemetry endpoints, and that non-deterministic scanning leaves windows of "days or weeks" where malicious modifications go undetected while clean audit badges keep displaying. Mitiga's advice is equally direct: don't trust skills based on stars or upvotes, which adversaries can manufacture with bots and fake accounts. Popularity is not verification — and any directory that blends the two is asking you to trust signals that are trivially spoofable.

This is the blueprint SkillzBench is being built around. Every skill version is pinned to a commit SHA, license-checked, security-scanned, and quality-judged; every skill is benchmarked in a sandbox with the same model, with and without the skill, and the resulting Lift — measured in percentage points — is always shown separately from the 0–100 trust score. Money never touches scores: the platform is free, funded by blog advertising that never influences listings or ratings. It's still pre-launch, but the design principle is already fixed: show your work, separate performance from trust, and make the verification itself auditable.

Because "who verifies the verifiers?" has only one durable answer: verifiers whose methods you can inspect.

Your Action Plan: How to Vet Agent Skills Before You Equip Them

Security researchers now agree on an uncomfortable conclusion: the answer to "who verifies the verifiers?" is, currently, you. Every source in this research converges on the same finding — AI skill directories operate as unvetted aggregators, and the verification layer simply does not exist yet.

Treat every agent skill as untrusted code, because that is exactly what it is. A large-scale analysis of 31,132 skills found that 26.1% contain at least one vulnerability, and skills bundling executable scripts are 2.12× more likely to carry one. Meanwhile, security researchers warn that stars, upvotes, and install counts can be artificially inflated, so popularity is not evidence of safety.

Here is the action plan, for managers and developers alike:

• Demand provenance. Pin every skill to a specific commit SHA, mirroring the trusted-publishing models of npm and PyPI that AI registries still lack.
• Prefer behavioral test results over popularity metrics. Static scanning flagged 26.1% of skills as risky, yet behavioral verification of 98,380 skills confirmed only 157 as genuinely malicious — a 150× precision improvement.
• Close the enterprise blind spot. If you audit software dependencies but not what your agents actually consume, you carry third-party risk you remain responsible for.
• Control what your agents see. Restrict skills and tools per agent rather than granting blanket access.

This is precisely the gap SkillzBench is being built to close. Its Verify pillar pins every version to a commit SHA, license-checks, security-scans, and quality-judges each skill; its Benchmark pillar tests every skill in a sandbox with the same model, with and without the skill, publishing the Lift in percentage points separately from ratings — money never touches scores. The free per-agent loadouts and free MCP connector mean your agents only ever see what you explicitly choose. SkillzBench is pre-launch, with a live waitlist, and everything on the platform will be free.

Skills, proven.

Verifying the Verifier: Transparency as the Final Trust Check

If a directory asks you to trust its ratings, the fair follow-up is simple: who audits the auditor? The uncomfortable answer from the research is that most AI skill registries provide minimal provenance verification compared with established package ecosystems like npm and PyPI, according to an analysis of AI skills security. Verification only counts when the verifier itself is auditable.

That means demanding three things from any directory. First, a published scoring methodology, not a mystery badge. Second, security and quality signals kept separate from performance claims — because install counts, stars, and upvotes can be inflated or spoofed, as Mitiga's supply-chain research warns. Third, a clear firewall between revenue and ratings, so money never touches a score.

The evidence justifies that skepticism. Static scanning flagged 26.1% of 31,132 analyzed skills as containing at least one vulnerability, per a large-scale study of agent skills, yet behavioral verification of 98,380 skills confirmed only 157 as malicious — a reminder that flag rates and confirmed malice are different things. Meanwhile, a single actor drove 54.1% of confirmed malicious skills through templated brand impersonation, according to follow-up behavioral research.

SkillzBench is being built around exactly these principles:

• A 0–100 trust score assembled from transparent security, quality, license, maintenance, and community signals
• Benchmark Lift published separately from ratings, measured in a sandbox with the same model, with and without each skill
• Advertising revenue that never influences scores or listings — everything stays free for users
• Continuous re-scanning and re-benchmarking as upstream skills change, with alerts when they do

Transparency is the final trust check because it makes the verifier verifiable too. SkillzBench is pre-launch and actively being built, and the waitlist is open — join it to follow the methodology as it ships.