From Zero to Eval: Building a Benchmark for Agent Skills

The Skill Supply Chain Nobody Is Testing

Every agent skill you install is a set of instructions that gets blended into your agent's context — and almost nobody is checking what those instructions actually say. Skills have spread across registries, GitHub repos, and community collections at remarkable speed, but evaluation infrastructure never caught up.

The numbers make the risk concrete. A Snyk analysis found that 13.4% of skills on ClawHub and skills.sh — roughly 534 of 3,984 — carried at least one critical security issue, including malware distribution, prompt injection attacks, and exposed secrets.

Worse, the problem isn't confined to executable code. University of Maryland researchers demonstrated that semantic edits as short as 20 tokens to a skill's description can manipulate how a skill is discovered in a registry, whether an agent selects it over alternatives, and whether it passes governance checks — with attack success rates reaching 86%. Those same edits evaded registry scanning defenses between 36.5% and 100% of the time.

The core insight from that research is uncomfortable: skill text is a security-sensitive object that traditional code scanners are structurally blind to. As the researchers put it, an attacker may not need to hide malware in executable code at all — small changes to natural-language descriptions are enough to steer agent behavior. Skills effectively function as user-authorized prompt injection, since their text is combined directly with user input and system prompts.

Existing registries have documented blind spots too. In ClawHub-style review, only the first 10,000 characters of a long SKILL.md file get passed to the LLM reviewer — an evasion vector attackers can exploit by simply padding content.

What's missing is a layer that treats skills the way package ecosystems treat dependencies: continuously discovered, version-pinned, scanned in full, and tested against real tasks. That's the gap SkillzBench is being built to fill. Its Verify pillar is designed to scan the entire skill text — not a truncated prefix — including checks for prompt injection and semantic manipulation, alongside license validation and version pinning to a commit SHA. Its Benchmark pillar tests each skill in a sandbox with the same model, with and without the skill, so the Lift in percentage points is measured rather than assumed.

The supply chain for agent skills is already global and growing. What it lacks is anyone systematically checking the goods moving through it. Building that checking layer — before trust has to be earned the hard way — is the problem worth solving now.

Join the SkillzBench waitlist to follow the build and get early access when benchmarking goes live.

What a Real Skill Benchmark Looks Like: The With/Without-Skill Method

The question every skill author eventually asks is simple: does my skill actually help, or does it just add tokens to the prompt? Until recently, no reusable methodology existed for answering it. A peer-reviewed framework from Tessl researchers now provides one — and its design is becoming the template for rigorous skill evaluation.

The methodology is elegantly blunt. The researchers evaluated 500 real-world open-source skills drawn from roughly 100 organizations — including Anthropic, Google, ElevenLabs, and Shopify — across about 1,000 realistic tasks and 19 agent–model configurations. The critical design choice: every task is solved twice, once with the skill available and once without. The difference between the two runs is the skill's marginal value, measured rather than assumed.

This with/without-skill design answers the author's question directly: given a newly created skill, how do you know it improves performance on the tasks it's meant to support? The study found that access to a skill significantly changes model behavior compared to a no-skill setup — and that models vary widely in how closely they follow skill instructions, producing substantial differences in gains. A skill that lifts one model may do little for another, which is why testing across multiple agent–model configurations matters.

Rigor also depends on how you grade. Anthropic's eval guidance — well, their actual guidance at demystifying evals — distinguishes what an agent claims from what actually happened: a flight-booking agent may say "your flight is booked," but the outcome is whether a reservation exists in the database. Their grader taxonomy spans three types:

• Code-based graders: fast, cheap, and objective, but brittle to unexpected solutions.
• Model-based graders: flexible and nuanced, but non-deterministic and requiring human calibration.
• Human graders: the gold standard, but expensive and slow.

Anthropic also separates capability evals, which should start with low pass rates, from regression evals, which should sit near 100% — with high-pass capability tests eventually graduating into regression suites. The Tessl framework applies similar depth to scoring, using two rubric sets per task (goal-completion and instruction-following), each scored 1–10 and combined into a 100-point total rather than a binary pass/fail. That matters, because Princeton's SAGE group argues that pass/fail benchmarks lose critical information that only log analysis can recover.

This is precisely the approach SkillzBench is building around its planned Lift metric: test every skill in a sandbox with the same model, run each task with and without the skill, and publish the difference in percentage points. Lift is shown separately from trust ratings, so a skill's measured impact is never blended into an opinion. The methodology is validated; the work now is applying it continuously and independently across the skill ecosystem.

Why Pass/Fail Isn't Enough: Rubrics, Logs, and the Lift Metric

A benchmark that says "pass" tells you almost nothing. What did the agent do to get there? Did it follow the skill's instructions, or ignore them and luck into the right answer? Princeton's SAGE research group puts it bluntly: agent benchmarks reporting only pass/fail outcomes lose critical information, and systematic log analysis is necessary to assess internal validity, external validity, and safety.

This is why rubric-based scoring matters. The Tessl evaluation framework — which tested 500 real-world skills across roughly 1,000 tasks and 19 agent-model configurations — scores each task on two rubric sets: goal-completion and instruction-following, each rated 1–10 and summed to 100. That distinction is everything. An agent can complete a goal while completely ignoring the workflow the skill encodes, which means the skill added nothing.

The deeper question any skill author eventually asks: does this skill actually help? The Tessl framework answers it by solving every task twice — once with the skill, once without — to measure marginal value. SkillzBench is building its Benchmark pillar on the same principle: run each skill in a sandbox with the same model, with and without it, and publish the difference as a Lift metric in percentage points.

Keeping Lift separate from trust ratings is a deliberate design choice. A skill can be well-maintained, cleanly licensed, and secure — and still make an agent slower or less accurate. Conversely, a risky skill might deliver real performance gains. Collapsing those dimensions into a single score would hide exactly the tradeoffs users need to see.

• Pass/fail outcomes discard how the agent reached its answer
• Rubrics capture both goal completion and instruction adherence
• With/without comparison isolates the skill's marginal contribution
• Separate trust and Lift scores prevent conflating safety with usefulness

There's one more complication: the jagged frontier. The Stanford AI Index notes that a top model can win a gold medal at the International Mathematical Olympiad yet read an analog clock correctly just 50.1% of the time. The Tessl researchers found the same unevenness with skills — models vary widely in how closely they adhere to skill instructions, producing substantial differences in performance gains across configurations.

That variance is why a single number, measured once, can't settle the question. A skill that lifts one model considerably might do nothing for another. Continuous, same-model, with-and-without measurement — the approach SkillzBench is designing for every skill it benchmarks — is the only honest way to answer whether a skill earns its place in your agent's context window.

Skills, proven — join the SkillzBench waitlist to see benchmark results as they publish.

Benchmarks Decay: Staying Current in a Saturated Market

A benchmark is a snapshot of a moment in time — and in agentic AI, moments pass fast. SWE-bench Verified scores climbed from 60% to near 100% in a single year, and agent task success on OSWorld jumped from 12% to roughly 66% over a comparable period, according to the Stanford HAI AI Index. Any evaluation built today starts decaying the day it ships.

This saturation problem is well documented. Princeton's SAGE group has studied what benchmarks can still tell us once top scores approach the ceiling, arguing that pass/fail outcomes alone lose critical information and that systematic log analysis is needed to assess validity and safety (https://sage.cs.princeton.edu/). Anthropic's engineering guidance makes the same point from the practitioner side: capability evals should start at a low pass rate, and once they approach 100%, they should "graduate" into regression suites (https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents). A benchmark that everyone passes measures nothing.

For skill evaluation, decay compounds. The underlying models change monthly, agents change their instruction-following behavior, and skill authors push upstream changes that can alter results entirely. The Tessl skill-evaluation framework found that models vary widely in how closely they adhere to instructions encoded in skills — meaning a Lift score measured against one model version may not hold against the next.

Designing for decay means building re-evaluation into the pipeline from day one, not bolting it on later. A durable approach includes:

• Tracking upstream skill changes and re-pinning versions to commit SHAs when authors update them
• Re-scanning skill text for security issues, since semantic edits as small as 20 tokens can manipulate discovery and selection with up to 86% success (https://www.theregister.com/ai-ml/2026/05/22/minor-edits-to-ai-skills-can-make-agents-go-rogue/5245413)
• Re-benchmarking with and without each skill against current models, so Lift reflects today's baseline, not last quarter's
• Notifying users when a re-run changes a score they relied on

The stakes are enterprise-sized. Only 21% of companies planning agentic AI have mature governance models, per Deloitte research, and Gartner predicts a quarter of enterprise breaches will trace to AI agent abuse by 2028. Teams without continuous re-evaluation are making deployment decisions against stale evidence in a market that moves annually.

This is why SkillzBench treats "Stay current" as a core pillar rather than a maintenance afterthought — tracking upstream changes, re-scanning, and re-benchmarking continuously, so a skill's Lift and trust score reflect the present, not the past.

ctaText: See how SkillzBench keeps skill evaluations current — join the waitlist at skillzbench.com. socialProofText: Built by AIQ Labs' independent testing lab — skills, proven.

Putting It Together: From Benchmark to Per-Agent Loadout

Everything upstream — discovery, verification, benchmarking — only matters if it changes what your agent actually loads at runtime. That's the final step: turning a continuous benchmark pipeline into a per-agent loadout where the agent sees nothing you haven't vetted.

The mechanics are deliberately simple. Every skill is pinned to a commit SHA, so what you evaluated is exactly what you install — no silent upstream edits. Security scanning runs over the full skill text, not the first 10,000 characters; that matters because researchers showed attackers can hide malicious triggers beyond the truncation point, where a registry's LLM reviewer never sees them. Semantic edits as short as 20 tokens achieved up to 86% success at steering skill discovery in University of Maryland experiments, so text itself gets treated as a security-sensitive object, not just code.

Then comes the separation that keeps scores honest. The trust score (0–100, built from security, quality, license, maintenance, and community signals) lives apart from Lift — the measured performance delta in percentage points from running each task with and without the skill, the same design a peer-reviewed framework validated across 500 skills and ~1,000 tasks. A skill can score well on one and poorly on the other. You see both, and you decide.

That decision is where per-agent loadouts come in. One MCP server, and for each agent you toggle which skills and tools it can access:

• The agent only sees skills you've explicitly enabled — nothing from the broader ecosystem leaks in.
• Trust scores and Lift sit next to each toggle, so approval is an informed click, not a leap of faith.
• Upstream changes trigger re-scans and re-benchmarks, with notifications when a pinned skill drifts.

This maps directly to what the market says works. Precedence Research frames the workable commercial model as "bounded autonomy with human approval gates" — and notably, 75% of organizations are piloting or deploying agents while only 15% go fully autonomous. Only 21% of companies planning agentic AI have mature governance models, which is precisely the gap a vetted loadout closes.

SkillzBench is building this now: the continuous pipeline, the full-text scanning, the loadout controls. The waitlist is open if you want skills, proven, before your next agent ships.