Why Skill Quality Judging Needs Structured Outputs

The Hidden Threats Lurking in Unstructured Skill Files

You cannot tell a safe skill from a risky one by reading it. The threats that matter most are the ones designed to survive a human skim — and they are already circulating at scale.

A large-scale security study of more than 50,000 skills found that over 26% carry at least one security vulnerability across 14 distinct attack patterns. A separate audit of nearly 4,000 skills classified 13.4% as containing critical-level issues and confirmed 76 malicious payloads built for credential theft and data exfiltration.

The problem is not that people are careless. It is that the SKILL.md format itself splits human-readable instructions from executable scripts, so metadata can look perfectly innocent while the real payload lives elsewhere. Malicious instructions have been found embedded in HTML comments and invisible Unicode tag codepoints — content that is functionally invisible during a standard manual file review.

Here is where the real danger concentrates:

• Invisible Unicode smuggled instructions that render as whitespace but steer agent behavior
• HTML comments carrying prompt injections that never appear in a rendered preview
• Bundled scripts with `curl | bash` installers or hardcoded secrets hidden from the markdown body
• Prompt injection paired with malicious code — a combination found in 91% of confirmed malicious skills

That last point deserves emphasis. Every confirmed malicious skill in the Snyk corpus contained malicious code patterns, and 91% simultaneously used prompt injection to prime the agent into executing what its safety rails would normally reject. This dual-threat vector has no real equivalent in traditional package ecosystems.

Manual review cannot keep pace with this. Daily skill submissions jumped from under 50 to over 500 in just a few weeks, and most users simply lack a reliable way to verify what a skill does before running it. Static analysis alone also falls short, since it misses time-delayed payloads and natural-language instructions that are semantically malicious but syntactically clean.

This is exactly the gap structured, machine-readable verification is built to close. Deterministic scanners achieve 90–100% recall on confirmed malicious skills with zero false positives on top legitimate packages. SkillzBench is building this kind of multi-tier pipeline — schema validation, secret detection, license checks, and behavioral scanning — so that hidden threats surface before they ever reach your agent.

Why Human Review and Free-Text Judging Fail at Scale

If you've ever skimmed a skill's README before installing it, here's the uncomfortable truth: that skim tells you almost nothing. The SKILL.md format separates friendly, human-readable instructions from executable scripts, which means malicious logic can sit in a bundled script while the metadata looks perfectly innocent. Researchers have even found hidden instructions embedded in HTML comments and invisible Unicode tag codepoints — text that's literally invisible during a normal file review, according to a Mondoo security analysis.

The scale of the problem makes ad-hoc checking untenable. In the largest public audit to date, Snyk scanned 3,984 skills and found that 13.4% contained at least one critical-level security issue, while 36.8% had at least one flaw at any severity. The researchers put it bluntly: this isn't theoretical risk, it's an ecosystem already under attack — and if you installed a skill in the past month, there's roughly a 13% chance it carries a critical flaw, per Snyk's toxic skills research.

What makes these attacks so hard to catch by eye is how they combine techniques:

• 100% of confirmed malicious skills contain malicious code patterns, and 91% also employ prompt injection — priming the agent to accept code that safety mechanisms would normally reject.
• Instructions can be semantically malicious but syntactically clean, so static analysis and casual reading both miss them.
• Time-delayed payloads activate long after installation, defeating one-time review entirely.
• 157 confirmed malicious skills averaged four distinct attack vectors each, meaning no single check catches them.

Human review fails for a simple reason: nobody can actually verify what a skill does before running it, and the volume makes it worse. Daily skill submissions on one hub jumped from under 50 to over 500 in a few weeks — no review team can keep pace with that.

Free-text LLM judging fails for a different reason: it produces prose, not data. When a judge model returns paragraphs of evaluation, results can't be parsed, aggregated, tracked over time, or reproduced across thousands of skills. As Philipp Schmid advises in his skill testing guide, you should "use structured output to constrain the LLM's response to a typed schema so that results become parseable and trackable." Without that constraint, every judgment is a one-off opinion.

The contrast with structured approaches is stark. Deterministic detectors in structured scanning engines achieve 90–100% recall on confirmed malicious skills with 0% false positives on legitimate ones — numbers no human reviewer or free-text judge can match at scale. That's why SkillzBench is building its quality judging around structured, machine-readable verification: typed schemas for LLM judgments, deterministic security checks first, and sandboxed benchmarking to measure each skill's actual Lift. For a directory meant to be trusted, structure isn't a preference — it's the safety mechanism.

The Structured Solution: Typed Schemas, Deterministic Checks, and A/B Benchmarks

The unstructured mess of free-text reviews and ad-hoc checklists cannot catch what attackers are actually hiding. A research-backed, three-layer evaluation pipeline offers the fix.

The first layer enforces typed JSON schemas for every LLM-as-judge response. Philipp Schmid recommends this approach explicitly, noting that constraining an LLM's output to a defined schema makes results parseable and trackable across thousands of skills. Free-text evaluations drift; structured outputs aggregate into consistent scores.

The second layer applies deterministic regex checks before any expensive LLM call. Snyk's mcp-scan engine demonstrates the payoff: CRITICAL-level detectors achieve 90–100% recall on confirmed malicious skills with 0% false positives against the top 100 legitimate skills. Deterministic checks are fast, cheap, and catch the patterns that matter most, like hardcoded secrets, curl | bash chains, and known prompt injection markers. LLMs should be reserved for qualitative dimensions (code structure, instruction clarity) that regex genuinely cannot capture.

The third layer runs sandboxed A/B benchmarks to measure real Skill Lift. NVIDIA's testing showed correctness climbing from 46 to 87, a +41-point jump, when agents used verified skills versus baseline. Effectiveness rose from 39 to 78 (+39 points). These numbers are not theoretical; they come from running the same model against the same tasks with and without the skill attached.

A practical benchmark design includes:

• 10–20 prompts as a starting evaluation set per skill
• 3–5 trials per prompt to absorb agent nondeterminism
• Outcome-based grading (did it work?) rather than path-based grading (did it follow the expected steps?)
• Relevance checks, since a skill loaded at the wrong moment can degrade performance

Token efficiency deserves measurement too. NVIDIA found one skill cut tokens by 76.9% (from 617,306 to 142,540), while another increased consumption by 120.3%, proving savings are not automatic. A structured evaluation tier is the only way to surface that variance before it hits production costs.

For platforms building skill directories, this layered approach is the differentiator. SkillzBench is constructing exactly this pipeline: deterministic security scans first, structured LLM judging second, sandboxed A/B benchmarks third, with Lift reported separately from any trust rating and ad revenue never touching the numbers. Skills, proven through evidence rather than vibes.

How to Evaluate Skills Safely: A Practical Checklist

Installing a random skill from the internet is a bit like running a random script from a stranger's repository — except the threat can hide in invisible Unicode characters that no human reviewer will ever see. A practical checklist turns that gamble into a repeatable process anyone can follow.

Start with pinning every skill to a commit SHA. The barrier to publishing a skill is just a SKILL.md file and a GitHub account, with no default code signing or security review — so a version that was safe last week may not be the version you get today. Pinning means you always know exactly which code your agent is executing.

Next, look for evidence of security scanning and license checks before you trust a skill. The numbers explain why: a large-scale audit found that 13.4% of scanned skills contain at least one critical security issue, and roughly 37% have at least one flaw at any severity. Malicious instructions have even been found hidden in HTML comments and invisible Unicode tag codepoints, which security research shows manual file review simply cannot catch.

Then demand a measured Skill Lift — the percentage-point improvement an agent shows with the skill versus without it. NVIDIA's SkillEvaluator work found correctness jumped from 46 to 87 with verified skills, but also that one skill cut token usage by 76.9% while another increased it by 120.3%. Savings are not automatic; they require measurement against a baseline, typically with 3-5 trials per test case to account for agent nondeterminism.

When judging results, grade outcomes, not paths. An agent that reaches the correct answer through an unexpected route has still succeeded — penalizing creative paths produces false negatives. Philipp Schmid, who recommends structured, typed-schema outputs for any LLM-based judging so results stay "parseable and trackable," also suggests 10-20 prompts as a starting evaluation set for a single skill.

Finally, be wary of unverified registries. Daily skill submissions jumped from under 50 to over 500 in a few weeks, and 100% of confirmed malicious skills contained malicious code patterns while 91% also used prompt injection — a combination that primes agents to execute code safety mechanisms would normally reject. If a registry can't show you scanning, pinning, and benchmark methodology, treat every listing as untrusted.

The checklist in brief:

• Pin the skill to a specific commit SHA so you know exactly what your agent runs.
• Look for security scanning, license checks, and validation — not just a human-readable description.
• Demand a measured Lift: same model, with and without the skill, 3-5 trials per case.
• Grade outcomes, not the path the agent took to get there.
• Avoid unverified registries that can't explain their verification process.

This multi-tier pipeline — static validation, quality judging, then sandboxed A/B benchmarking — is exactly what SkillzBench is building: a free, independent directory that pins every skill version to a commit SHA, security-scans and license-checks it, benchmarks it in a sandbox with and without the skill, and publishes the Lift in percentage points separately from its 0-100 Trust Score. Money never touches the scores, and everything on the platform stays free. If you want skills, proven — join the waitlist to be first in line when it launches.

What This Means for the Future of the Skill Ecosystem

The agent skill ecosystem today looks a lot like npm and PyPI in their early years: a flood of packages, almost no security review, and an open door for anyone with a GitHub account and a SKILL.md file. Snyk researchers describe it bluntly — daily skill submissions jumped from under 50 to over 500 in a matter of weeks, and the ecosystem is "already under attack." In their audit of nearly 4,000 skills, 13.4% contained at least one critical security issue, and 76 confirmed malicious payloads were built for credential theft and data exfiltration.

The industry's answer is a shift from manual review toward structured, automated verification. Static analysis alone can't catch threats like invisible Unicode tag codepoints or time-delayed payloads hidden in bundled scripts, which is why security teams are combining deterministic checks with multi-model behavioral analysis. Structured scanning engines are already achieving 90-100% recall on confirmed malicious skills with 0% false positives on legitimate ones — numbers no human review process can match at scale.

That same structured approach is what makes honest quality measurement possible. NVIDIA's SkillEvaluator work shows that verified, structured skill definitions can lift correctness from 46 to 87 and effectiveness from 39 to 78, while token efficiency swings wildly between skills — one cut token use by 76.9%, another increased it by 120.3%. Without structured benchmarking, you simply can't tell which skills help and which quietly cost you money.

This is the future SkillzBench is building toward. The free directory will continuously crawl, dedupe, and verify skills from GitHub — pinning every version to a commit SHA, license-checking, security-scanning, and quality-judging each one. Every skill will also be benchmarked in a sandbox with the same model, with and without the skill, so the Lift is published in percentage points rather than marketing claims.

Two design choices matter most:

• Trust Score and Lift stay separate. Security, quality, license, maintenance, and community signals feed a 0-100 Trust Score; measured performance is always shown alongside, never blended in.
• Money never touches scores. The platform is free — directory, MCP connector, benchmarks, and alerts — with revenue from blog advertising only, so no payment or ad spend can influence a rating or listing.
• Structured outputs everywhere. LLM-based judging is constrained to typed schemas so results stay parseable, trackable, and reproducible — the approach practitioner Philipp Schmid recommends for making LLM-as-judge grading reliable.

If you're adopting agent skills for your team or your agents, the takeaway is simple: treat unverified skills the way you'd treat an unsigned npm package from a stranger. Join the SkillzBench waitlist to get early access to the free directory and the MCP connector that lets you equip any agent with only the skills you choose — verified, benchmarked, and scored.