Question 1

How does SkillzBench's benchmark-as-a-service differ from a traditional benchmarking study like the Schwab RIA benchmarking study?

Accepted Answer

Traditional studies like the 2025 RIA Benchmarking Study rely on self-reported data from participating firms, which carries inherent bias risk. SkillzBench instead runs controlled sandbox tests — with and without the variable under test, on the same model — and publishes results as a percentage-point Lift. Inputs are pinned to commit SHAs and independently validated before benchmarking, and Lift is always shown separately from ratings.

Question 2

What exactly gets benchmarked, and what does 'Lift' mean in my results?

Accepted Answer

SkillzBench benchmarks AI agent skills — SKILL.md instruction packages used with Claude Code, Codex, Cursor, Gemini CLI, and chat-app agents. At launch the scope is instruction-only skills; script-bearing skills are listed link-out only. Lift is the percentage-point improvement measured by running the same model with and without the skill in a sandbox, isolating the skill's true effect rather than a correlational estimate.

Question 3

Can I trust benchmark results when published research says ~of skills carry vulnerabilities?

Accepted Answer

That published finding — roughly of skills carrying at least one vulnerability — is exactly why SkillzBench validates before benchmarking. Every skill is pinned to a commit SHA, validated, license-checked, and security-scanned, and each carries a 0-100 trust score built from security, quality, license, maintenance, and community signals. Risky inputs are flagged before they ever reach a benchmark run.

Question 4

SkillzBench is pre-launch — what does the timeline look like for an agency joining now?

Accepted Answer

SkillzBench is actively being built, and the waitlist and newsletter are live today. Joining now reserves your agency's position for priority benchmark runs as the service comes online, plus early access to the MCP server and per-agent loadouts. You'll receive build updates through the newsletter so you know exactly when priority runs open up.

Question 5

How does this compare to IT benchmarking services or outsourcing contract benchmarks from firms like ISG?

Accepted Answer

IT benchmarking services and outsourcing contract benchmarks typically assess cost and price data across large contract databases — valuable, but retrospective and dependent on data normalization choices. SkillzBench focuses on forward-looking, causal performance benchmarking for AI agent skills, with immutable version pinning and continuous upstream re-benchmarking. The two are complementary: ISG-style price benchmarking for contracts, SkillzBench for proving which agent skills actually help.

Benchmark-as-a-Service for Agencies