Question 1

How does SkillzBench measure whether an agent skill actually helps?

Accepted Answer

SkillzBench tests every skill in a sandbox using the same model, running tasks both with and without the skill, then publishes the difference as Lift in percentage points. Lift is always shown separately from ratings, and money never touches scores — so the number reflects measurement, not sponsorship.

Question 2

What kinds of skills does SkillzBench support at launch?

Accepted Answer

Launch scope is instruction-only skills — SKILL.md packages plus text references. Script-bearing skills are listed link-out only, so you always know whether you're installing instructions or executable code.

Question 3

Why can't I just trust public agent leaderboards and vendor benchmarks?

Accepted Answer

Research found severe issues in 8 of 10 popular AI agent benchmarks, causing up to 100% misestimation of capabilities, and τ-bench scored a do-nothing agent as correct on 38% of airline tasks. Scores are also scaffold-dependent, so absolute numbers degrade across frameworks. Independent, controlled testing is the only reliable way to compare skills.

Question 4

When can I start using SkillzBench, and what does it cost?

Accepted Answer

SkillzBench is pre-launch and actively being built — the waitlist and newsletter are live now. The pricing hypothesis includes a generous free tier, a Pro plan around $9-15/mo for more agents and unlimited enabled skills, Teams around $15-25/seat/mo for org loadouts, allowlists and audit, and a Creator plan around $5-10/mo for verified authors.

Question 5

How is SkillzBench different from browsing GitHub for skills myself?

Accepted Answer

GitHub gives you no verification: no security scan, no license check, no version pinning, and no measurement of whether the skill actually improves your agent. SkillzBench crawls and dedupes skills, pins each version to a commit SHA, quality-judges and security-scans them, benchmarks the Lift, and tracks upstream changes with re-scan notifications.

Agent Skill Benchmarking for Agencies