Benchmarks vs Ratings: Keeping Money Away From Scores

Broken Trust Signals: Why Ratings and Install Counts Get Gamed

You install a skill because it has thousands of downloads and a clean rating. Your agent exfiltrates your credentials anyway. That gap between what the marketplace shows and what's actually true is the core problem with popularity-driven skill discovery.

The warning signs aren't hypothetical. Orca Security researchers demonstrated that install counts on a skill marketplace could be inflated with a single unauthenticated HTTP request — no rate limiting, no token validation. They pushed a malicious skill to "several thousand" installs, landed it on the marketplace homepage, and real users installed it before anyone flagged it.

The trust signals around that skill were equally hollow. Security scans ran only at skill creation and again when a popularity threshold was crossed, leaving a window where malicious modifications went undetected while the platform kept displaying the original clean audit results. As Orca put it: "Don't rely solely on platform-provided security badges or install counts."

ClawHub tells a similar story. Publishing there required only a SKILL.md file and a week-old GitHub account — no code signing, no security review. Snyk's researchers found 76 confirmed malicious payloads on the platform, and 8 malicious skills were still publicly available at publication. Their summary is blunt: "The user installed a popular, well-reviewed skill. Yet the agent is compromised."

The baseline risk makes these broken signals worse, not better. A large-scale academic study found 26.1% of analyzed skills contain at least one vulnerability, with 13.3% enabling data exfiltration. Snyk's independent scan put critical-severity issues at 13.4% of skills — meaning a roughly one-in-eight chance that a recently installed skill carries a critical flaw.

What makes a marketplace's ratings dangerous isn't just that they're wrong. It's that they reward exactly the behaviors that create risk:

• Install counts are cheap to fake, so attackers buy visibility that real authors have to earn slowly.
• Reviews measure user sentiment, not whether a skill secretly fetches poisoned third-party content through indirect prompt injection.
• Popularity thresholds can trigger re-scans — which means gaming popularity also changes when a skill gets scrutinized.
• No marketplace in the ecosystem currently publishes anything like a percentage of skills security-audited, so there's no baseline to compare ratings against.

This is why SkillzBench treats measured performance and trust as separate, non-purchasable signals. Its Lift metric — computed by running the same agent with and without a skill in a sandbox — follows the approach NVIDIA's SkillEvaluator validated, and its 0–100 trust score draws on security scans, version pinning to commit SHAs, and license checks rather than popularity. Neither number can be bought, boosted, or bundled into placement.

Ratings tell you what people felt. Benchmarks tell you what happened. When one in four skills carries a vulnerability, that difference decides whether your agent helps you or leaks for you.

Measured Lift: The Benchmark Alternative to Subjective Ratings

If a rating tells you a skill is "great," you still know nothing. A benchmark that runs the same agent twice — once with the skill, once without — tells you exactly how many points changed.

That's the shift underway in agent-skill evaluation. NVIDIA's SkillEvaluator runs agents in isolated sandboxes, with and without each skill, and reports the difference as "Skill Lift" in points. Across 300+ verified skills spanning 30+ products, NVIDIA measured average lifts of roughly +35 to +41 points — a controlled A/B measurement, not an opinion.

The academic side is converging on the same principle with even stricter methodology. SkillsBench treats agent skills as first-class evaluation artifacts and replaces LLM-as-a-judge scoring with deterministic verifiers — programmatic assertions that produce reproducible pass/fail results without judge variance. It also enforces leakage prevention, blocking skills that embed task-specific filenames, constants, or test references, with a CI validation agent catching skill-solution leakage.

The headline numbers look encouraging, but the honest framing matters more:

• SkillsBench found curated skills deliver an average gain of 16.2 percentage points — but 16 of 84 tasks showed negative deltas.
• Per-domain lift ranged from +4.5 points (Software Engineering) to +51.9 points (Healthcare) — the domain mattered far more than the harness.
• NVIDIA explicitly cautions that scores show higher average performance on evaluated tasks, not pass probabilities — and that one skill cut tokens 76.9% while another increased them 120.3%.

In other words, a benchmark score is a measurement, not a guarantee. A skill that helps on average can still hurt your specific workload, and NVIDIA's own data shows lift varies far more across products (+2 to +46 points) than across evaluation harnesses (~5 points). As their authors put it, a skill can only be measured as precisely as its evaluation set describes the job.

This is why SkillzBench is building its Benchmark pillar around the same with/without methodology — same model, sandboxed runs, Lift published in percentage points — and always shown separately from trust ratings. The research makes clear that a measured differential and a trust score answer different questions, and blending them would corrupt both.

The deeper lesson from the data is epistemic humility. Different evaluation designs produce materially different headline numbers, and methodology transparency is what makes a score trustworthy, not the number itself. A benchmark you can inspect, reproduce, and argue with beats a five-star rating every time — and it's the only kind of signal that stays credible when money never touches it.

The Corruption Vector: What Happens When Money Meets Scores

No one has yet published a study proving that paid placement corrupts AI evaluation scores — but the warning is already written on the wall, in a language anyone can read.

Orca Security demonstrated that install counts on a skill marketplace could be inflated with a single unauthenticated HTTP request — no rate limiting, no signing, no token validation. The researchers pushed a malicious skill to "several thousand" installs, landing it on the marketplace homepage, where real users installed it before it was flagged. That is what happens when a trust signal costs nothing to fake.

The pattern generalizes. Any score that can be bought — directly through paid placement, or indirectly through popularity weighting — becomes a target for gaming rather than a measure of merit. Snyk's researchers describe the result plainly: "The user installed a popular, well-reviewed skill. Yet the agent is compromised." Their scan of ClawHub found 8 confirmed malicious skills still publicly available at publication, despite platform curation signals.

The stakes make manipulation worth the effort. Academic analysis of 31,132 skills found 26.1% carry at least one vulnerability, and 8.7% of ClawHub skills have direct access to financial accounts, trading platforms, or crypto operations, according to Snyk's research. A corrupted score isn't an inconvenience — it's a supply-chain attack vector with money on the other end.

The structural vulnerabilities Orca documented map precisely onto what a purchasable score would enable:

• Security scans ran only at skill creation and at popularity thresholds, leaving an exploitation window where malicious changes go undetected while clean audit badges stay displayed.
• Popularity thresholds themselves triggered re-scanning — meaning inflated counts didn't just deceive users, they rewired the platform's own security workflow.
• Same-name installs silently overwrote existing skills with no confirmation, diff, or version pinning, so a gamed listing could displace a legitimate one.

This is why SkillzBench's design commitment is structural rather than aspirational: Lift — the measured percentage-point difference from running an agent with and without a skill in a sandbox — is always shown separately from the 0–100 trust rating, and money never touches either. Paid tiers buy capacity and features, never scores, placement, or benchmark priority. The controlled with/without A/B methodology that the industry is converging on only stays credible if the ledger and the leaderboard never share a column.

A score you can buy is a score attackers can rent. The Orca proof-of-concept shows how cheaply unearned trust signals fall — and why keeping money out of measurement isn't a nice-to-have, it's the whole point.

Building a Score System Money Can't Buy: SkillzBench's Design Principles

Every trust signal that can be bought, faked, or inflated eventually will be. That's the uncomfortable lesson from the agent-skill ecosystem, and it's why SkillzBench is being built around structural safeguards rather than good intentions.

The design starts with version pinning. Orca Security's research showed that installing a skill with the same name as an existing one silently overwrites it — no confirmation, no diff, no warning — and that update commands refresh everything with no per-skill granularity. Pinning every version to a commit SHA closes that door: the score you see is the score for the exact code that was scanned.

Re-scanning closes another. Orca found that marketplace security scans ran only at skill creation and when popularity thresholds were crossed, leaving an exploitation window where malicious modifications went undetected while stale clean-audit badges stayed visible. Tracking upstream changes and re-scanning on every update is the only honest answer.

The core safeguards SkillzBench is building:

• Every skill version pinned to a commit SHA, so scores refer to exact code, not a moving target
• Re-scanning and re-benchmarking whenever upstream repositories change
• No popularity-weighted components in the trust score — install counts are trivially inflatable via a single unauthenticated request, as Orca demonstrated
• Published methodology and limitations alongside every score, following the transparency practice NVIDIA applies to its Skill Lift evaluations

Transparency matters because raw numbers mislead. Vulnerability rates for skills vary from 13.4% critical in Snyk's scan to 26.1% in a separate large-scale analysis — different corpora and detectors produce materially different headlines. One audit even found a ~78% false-positive rate from YARA-based scanners. A score without its methodology is marketing, not measurement.

Finally, scope. At launch, SkillzBench covers instruction-only skills — SKILL.md packages plus text references — while script-bearing skills get link-out listings only. The data justifies this boundary: an analysis of 31,132 skills found that skills bundling executable scripts are 2.12× more likely to contain vulnerabilities than instruction-only ones. Narrowing scope to the measurably safer category keeps the risk profile defensible.

None of this is complicated. It's just discipline: scores that describe a pinned artifact, re-checked when the artifact changes, stripped of gameable popularity signals, and published with their caveats. Money buys capacity and features — never a number.

What Paid Tiers Buy (and What They Never Will)

If a paid tier could change a score, the score would be worthless. That's the uncomfortable conclusion of a year in which researchers showed how easily skill-marketplace trust signals get gamed — and why SkillzBench's pricing design starts from the opposite assumption.

The evidence for caution is concrete. Orca Security demonstrated that install counts on a skill marketplace could be inflated with a single unauthenticated HTTP request, pushing a malicious skill onto the marketplace homepage where real users installed it before it was flagged. Publishing on ClawHub requires only a SKILL.md file and a week-old GitHub account — no code signing, no security review. When popularity signals are that cheap to fake, any monetization that touches them makes things worse, not better.

The credible alternative is measurement, not opinion. NVIDIA's SkillEvaluator runs agents with and without a skill in isolated sandboxes and reports the difference as Skill Lift in points, while SkillsBench uses deterministic verifiers instead of LLM-as-a-judge to ensure reproducible pass/fail results. These numbers only mean something if nobody can buy them. That's the line SkillzBench is building around: paid tiers buy capacity and tooling — never scores, never Lift numbers, never placement.

Here's what each planned tier actually purchases:

• Pro (~$9–15/mo): more agents and unlimited enabled skills — capacity, not credibility.
• Teams (~$15–25/seat/mo): organizational loadouts, allowlists, and audit trails for compliance needs.
• Creator (~$5–10/mo): verified author identity and publishing tools — verification of who you are, not how good your skill is.
• Benchmark-as-a-service: priority runs — you skip the queue, but the methodology and the sandbox stay identical.

Honesty about limits matters as much as the firewall itself. NVIDIA explicitly cautions that benchmark scores are not pass-probability estimates, and vulnerability statistics vary wildly by methodology — one study found 26.1% of skills carry at least one vulnerability, while a Snyk scan found 13.4% with critical issues, and an independent audit documented a ~78% false-positive rate from YARA-based MCP scanners. Scores without disclosed methodology mislead even when no money is involved.

Before trusting any skill marketplace or benchmark provider, ask:

• Can anyone pay to change a score, a Lift number, or placement? (The answer must be no.)
• Is the evaluation methodology published, including limitations and dataset scope?
• Are benchmarks run in sandboxes with the same model, with and without the skill?
• Are versions pinned to commit SHAs and re-scanned on upstream changes?
• Is the trust score free of popularity-weighted components?

If a vendor hesitates on any of these, treat the score as marketing. SkillzBench's waitlist is open for teams that want the firewall built in from day one.