Trust Scores for AI Components: A Proposal

The Trust Gap: One in Four Agent Skills Carries a Vulnerability

If you installed an agent skill last month, there's a one-in-four chance it carried a security flaw. The first large-scale empirical study of the ecosystem analyzed 31,132 skills and found that 26.1% contain at least one vulnerability, spanning prompt injection, data exfiltration, privilege escalation, and supply chain risks.

The numbers get worse as you look closer. Snyk's audit of 3,984 skills found 13.4% contain at least one critical security issue, with 76 confirmed malicious payloads. And when Mondoo scanned 58,983 skills, detecting 11,929 threats, only 20% rated clean.

What makes skills uniquely dangerous is what they inherit. A skill runs with full agent permissions — shell access, file system read/write, credentials in environment variables, and persistent memory — yet publishing requires little more than a SKILL.md file and a week-old GitHub account. There is no code signing, no security review, and no sandbox by default.

This isn't theoretical. Real attacks are already documented:

• A December 2025 "GIF Creator" skill delivered MedusaLocker ransomware to users who installed what looked like a harmless media tool.
• A coordinated campaign of 30+ malicious skills was documented in February 2026, marking the ecosystem's first known malware operation.
• A state-sponsored group weaponized Claude Code through malicious MCP servers, showing attackers treat skills as a viable intrusion vector.
• Mitiga demonstrated silent codebase exfiltration through a skill that required only four user interactions and left the audit log completely empty.

The growth compounds the risk. Daily skill submissions jumped roughly tenfold in weeks — from under 50 in mid-January 2026 to over 500 by early February. Meanwhile, script-bearing skills are 2.12× more likely to contain vulnerabilities than instruction-only ones, and 91% of confirmed malicious skills also employ prompt injection to manipulate the agent itself.

The structural problem is that nothing gates publishing. As researchers put it, this isn't a bug in one marketplace — it's an ecosystem already under attack, where every skill should be treated as untrusted code. That's the gap SkillzBench is being built to address: continuous security scanning, version pinning, and a 0–100 trust score so users can see which skills are safe before installing them, not after.

If you want to know which skills are actually safe to install, join the SkillzBench waitlist — we're building the trust layer the ecosystem is missing.

Why Every Signal You Trust Today Is Gameable

Every trust signal you currently lean on — stars, install counts, that reassuring "scanned" badge — has been empirically broken. Not in theory, but in documented, reproducible attacks.

Start with GitHub stars. In a widely reported experiment, security researchers published a deliberately fake skill that passed every security scanner tested, including offerings from Cisco and NVIDIA. It also inherited roughly 36,000 stars from a popular repository. As the researchers put it, none of the signals people rely on caught it — not the scanners, not the stars, not the open-source reputation.

Install counts fare no better. Orca Security demonstrated that download numbers can be inflated with a single unauthenticated HTTP request. Worse, Orca showed three complete attack flows — bait-and-switch, nested skill injection, and delayed weaponization — that all achieved persistent code execution on skills that had passed platform security audits.

Then there is the audit trail itself. Mitiga built a proof-of-concept where a skill silently exfiltrated an entire codebase after just four user interactions. The skill-audit.log was completely empty. Mitiga's advice is blunt: don't trust skills based on stars or upvotes, because bot-driven reputation inflation is trivially easy.

Even the signals that should be hardest to fake — vendor security badges — fail. Mondoo's scan of nearly 59,000 skills found that high-popularity, vendor-published skills still receive High risk ratings. Popularity, it turns out, is not a proxy for safety.

The three failure modes share a single structural root:

• Point-in-time scanning: a scanner checks a fixed package, but a skill can point to external URLs whose content gets rewritten after the scan passes.
• Unverifiable popularity: stars, upvotes, and install counts can all be manufactured at negligible cost.
• Inherited trust: skills ride the reputation of the repositories and platforms they attach themselves to.

Trail of Bits, which independently replicated the scanner bypasses, summarized the asymmetry: an attacker can keep tweaking a payload until it passes, while the scanner only ever sees one snapshot. The scan happens once, but the content a skill points to can change at any time afterward.

This is the gap any credible trust system has to close. It is why SkillzBench pins every skill version to an immutable commit SHA and treats re-scanning as a continuous process rather than a one-time gate — because the attacks above all exploit exactly the assumption that yesterday's verification still holds today.

What a Real Trust Score Looks Like: Design Principles That Hold Up

A fake skill with 36,000 inherited GitHub stars sailed past every security scanner it faced — proof that trust signals need a complete redesign, not a patch. The good news is that three independent efforts have already converged on what works.

NVIDIA's SkillEvaluator measures skills through live with-skill versus without-skill sandbox evaluation, scoring five dimensions from security to efficiency. Mondoo's pipeline layers static rules, ML prompt-injection classification, and LLM threat analysis across six stages. And an IETF Internet-Draft proposes a reproducible pass-rate formula spanning 55 metrics. Their shared lessons distill into four design principles.

Pin versions to immutable commit SHAs. The structural flaw behind every scanner bypass is that "the scan happens once, but the page a skill points the agent to can be rewritten at any time after," as The Hacker News reported. A trust score attached to a mutable package is a promise about code that may no longer exist. Continuous re-scanning on every upstream change — not point-in-time audits — is the only honest posture.

Exclude popularity from the score entirely. Orca demonstrated install counts can be inflated with a single unauthenticated HTTP request, and Mondoo found high-popularity vendor skills still flagged High risk. Stars measure attention, not safety.

Keep Lift strictly separate from trust. NVIDIA's data shows Skill Lift ranges from +2 to +46 points across products, and some skills increase token costs by 120.3%. A skill can be safe but useless, or effective but risky — conflating the two misleads users on both axes.

Weight the score by what a skill actually contains. The empirical risk gradient is stark:

• Script-bearing skills are 2.12× more likely to contain vulnerabilities than instruction-only skills, per the large-scale arXiv study of 31,132 skills
• 100% of confirmed malicious skills contained malicious code patterns, with 91% also employing prompt injection, per Snyk's audit
• Instruction-only SKILL.md packages carry meaningfully lower structural risk, making them the right full-support scope at launch

This is exactly the architecture SkillzBench is building: every version pinned to a commit SHA, re-scanned and re-benchmarked when upstream changes, with Lift published separately from the trust score and money never touching ratings. Script-bearing skills will be listed link-out only until the risk calculus justifies more.

None of this requires inventing new science. It requires refusing the shortcuts — popularity badges, one-time scans, blended scores — that attackers have already learned to game. Skills, proven.

The SkillzBench Proposal: Five Pillars, One Score

Every trust signal agents currently rely on has been proven gameable. A fake skill inherited roughly 36,000 GitHub stars and passed every security scanner it encountered, while install counts were shown to be inflatable with a single unauthenticated request. SkillzBench is being built on a different premise: trust must be measured, not assumed.

The design rests on five pillars. Discover continuously crawls and dedupes skills from GitHub, because publishing grew roughly 10x in weeks and nobody is keeping a clean registry. Verify pins every version to an immutable commit SHA, runs license checks, and applies layered security scanning — the layered approach follows precedents like Mondoo's six-layer pipeline, which catches injection patterns invisible to human review. Benchmark tests every skill in a sandbox with the same model, with and without the skill, and publishes the measured Lift in percentage points, adapting the with/without-skill methodology NVIDIA validated across 300+ skills. Equip delivers one MCP server plus per-agent loadouts, so an agent only sees the skills and tools its operator explicitly chose. Stay Current tracks upstream changes and triggers re-scanning and re-benchmarking — directly addressing the structural flaw that a scan happens once while a skill's external content can be rewritten at any time afterward.

At launch, full evaluation covers instruction-only skills (SKILL.md plus text references). Script-bearing skills are listed link-out only, a scope choice grounded in the data: script-bearing skills are 2.12× more likely to contain vulnerabilities than instruction-only ones, and 100% of confirmed malicious skills contained malicious code patterns.

The trust score itself is a 0–100 composite drawn from five signal families:

• Security — layered scanning results, re-run continuously rather than at a single point in time
• Quality — structural validation and instruction clarity
• License — terms that permit the use you actually intend
• Maintenance — upstream activity and change history
• Community — a deliberately minor component, given how trivially stars and upvotes are spoofed

One boundary matters more than the rest: money never touches scores. Paid tiers — planned across free, Pro, Teams, and Creator options — buy agents, loadouts, and organizational controls, never favorable ratings. Lift is likewise always displayed separately from the trust score, because a skill can be safe but useless, or effective but risky; NVIDIA's own data showed Lift varying from +2 to +46 points across products, with some skills actually increasing token costs.

No source in the current research landscape proposes a complete framework combining security, quality, license, maintenance, and community signals. That's the gap SkillzBench is building to close — and the waitlist is open for teams who want trust scores they can actually audit.

What You Can Do Today While the Ecosystem Catches Up

Until the ecosystem matures, treat every skill you install as untrusted code — because that's exactly what it is, as Orca Security puts it. A skill runs with roughly the authority of a user prompt, inheriting shell access, file system permissions, and credentials in environment variables. That's not a reason to avoid skills entirely, but it is a reason to adopt a few defensive habits now.

Here's what works today:

• Pin every skill to a specific commit. If a skill points to a mutable branch or external URL, its content can be rewritten after you install it — the structural flaw behind the documented scanner bypasses.
• Prefer instruction-only skills where possible. Script-bearing skills are 2.12× more likely to contain vulnerabilities than instruction-only ones, according to a large-scale study of 31,132 skills.
• Ignore stars, upvotes, and install counts as trust signals. Researchers demonstrated that install counts can be inflated with a single unauthenticated HTTP request, and a fake skill inherited roughly 36,000 GitHub stars while passing every scanner tested.
• Re-verify whenever upstream changes. A one-time clean scan means nothing if the repository is modified afterward — treat point-in-time audits as a snapshot, not a guarantee.
• Audit the skills you've already installed for hardcoded secrets, external URLs, and instructions that push the agent toward exfiltration-friendly behavior.

The popularity-signal problem deserves emphasis. Mitiga explicitly warns against trusting skills based on stars or upvotes, and Mondoo found that even high-popularity, vendor-published skills — including one with 3 million recorded popularity — still received High risk flags. Reputation is easy to manufacture and even easier to inherit.

The deeper issue is that none of this should require manual vigilance from every team. The ecosystem needs continuous re-scanning, version pinning to immutable commits, and measured performance data — the same rigor we apply to package managers and container registries. That's the gap SkillzBench is building to close: a benchmarking and scanning infrastructure that pins every skill version, re-verifies on upstream change, and publishes measured Lift from with-and-without-skill sandbox testing, always shown separately from security ratings.

SkillzBench is currently pre-launch and under active construction. If you want first access when the benchmarking, scanning, and MCP-based control panel go live, join the waitlist at skillzbench.com — early subscribers will get in before the general release.