Testing Skills Across Claude, GPT and Gemini

There Is No Best Model — Only the Best Model for What

You've asked the question a hundred times: which one should I use — Claude, GPT or Gemini? The honest answer, as a Berkeley I School comparison puts it, is not which model is best, but which model is best for what.

The benchmark evidence backs that up with a clear domain split. Aggregated data from LM Council shows Claude leading coding and long-horizon agentic work — Claude Opus 4.7 tops SWE-bench Verified at 83.5%, and Claude models dominate METR Time Horizons, where Claude Mythos Preview sustains 1,044.8 minutes of autonomous task duration versus 352.2 for GPT-5.2. GPT leads mathematics and PhD-level science reasoning, scoring a perfect 100% on OTIS Mock AIME and leading GPQA Diamond at 94.6%.

Gemini, meanwhile, leads knowledge breadth, multimodal tasks and agentic-game environments — Gemini 3.1 Pro Preview tops Humanity's Last Exam at 46.44%, and Gemini 3 Pro Preview leads the BALROG agentic benchmark at 58.1%.

• Coding and long-horizon agent work: Claude leads, with Terminal-Bench 2.0 at 90.2% and four of the top five spots in code optimization.
• Math and science reasoning: GPT leads — but the lead is tier-dependent. On FrontierMath Tier 4, the hardest research-level problems, Claude Fable 5 actually leads at 87.8% versus GPT-5.5 Pro's 78.0%.
• Breadth and multimodal tasks: Gemini leads, including visual physics (VPCT) at 91.0%.
• Speed and cost: different leaders entirely. The Artificial Analysis leaderboard shows Claude models holding the top intelligence positions while Gemini 3.5 Flash-Lite delivers 449 tokens/s at $0.19 — with an intelligence score of just 22.

The same nuance applies to real-world coding costs. Databricks' benchmark of coding agents on a multi-million-line codebase concluded that "only a mix of tools can provide frontier performance" — the quality-per-cost frontier spans OpenAI, Anthropic and open-source models, and a cheaper model tied a premium one on quality.

For anyone building agents, this fragmentation has a practical consequence: the same skill may behave differently on each model. That's why testing a skill with and without it, on the same model in a controlled sandbox, matters more than picking a "winner" — the approach SkillzBench is building to publish each skill's Lift in percentage points. Its free directory and MCP connector are on the way; join the waitlist to test skills across all three model families when it launches.

Social proof: Published research found ~26% of agent skills carry at least one vulnerability — SkillzBench is being built so you can verify, benchmark and see the Lift before you trust a skill. Skills, proven.

Why Public Benchmarks Mislead: Outcomes Hide Behavior

So you've picked your winner from a leaderboard. Here's the uncomfortable truth: that number may not mean what you think it means.

Recent evaluation research found that frontier agent capability was under-elicited by nearly 50% in one case study — the agents could do far more than their scores showed, but task errors and ambiguities in the benchmark hid it. The problem is structural. As the researchers put it, benchmarks tell you what an agent achieved; only logs reveal how and why. A coding agent that fails on a minor syntax error and one that fails by deleting an entire Git branch score identically on an outcome-only test, yet pose wildly different risks.

Public benchmarks have a second problem: contamination. When Databricks benchmarked coding agents on its own multi-million-line codebase, it found public benchmarks suffer from training-data contamination and simply don't reflect proprietary schemas, internal tools or domain constraints. High scores, they concluded, do not guarantee production reliability, safety or cost efficiency in real workflows.

There's a third trap that catches almost everyone: you're never testing a model in isolation. As Anthropic's engineering team explains, when you evaluate an agent, you're evaluating the harness and the model working together. The scaffolding around the model — how context is managed, which tools are exposed — can matter as much as the model itself. Databricks found harness choice alone produced cost differences greater than 2x at equal quality.

And if you're choosing based on price? Databricks found that the token price of a model is a poor indicator of actual end-to-end task costs. On their real codebase, a cheaper model (GLM 5.2 at $1.28 per task) statistically tied a premium one (Opus 4.8 at $1.94 per task) on quality — while Sonnet 5 consumed 1.9x more tokens than Opus to complete less work. Sticker price tells you almost nothing about what a task will actually cost.

For anyone testing agent skills, the same logic applies with force. The only way to know what a skill contributes is to test it in a controlled sandbox, with the same model, with and without the skill — and publish the difference. That's the approach SkillzBench is building: benchmark every skill under identical conditions, verify it (version-pinned, security-scanned, license-checked), and show the Lift in percentage points, separate from any rating. Given that published research found roughly 26% of skills carry at least one vulnerability, "did it pass the leaderboard" is not a safety check either.

The takeaway: treat leaderboards as a starting hypothesis, not a verdict. Measure behavior, not just outcomes — and measure cost per task, not per token.

Skills Made It Worse — and Better: The Cross-Model Standard Nobody Vetted

A framework that started inside one vendor's environment now runs on all three major model families — and almost nobody has checked what it actually does. The skills standard began in Anthropic's Claude environment before being published as an open standard in December 2025, then adopted by OpenAI, Cursor and GitHub Copilot, according to security research from HiddenLayer. That makes skills a cross-model phenomenon: the same SKILL.md package can shape behavior in Claude, GPT and Gemini agents alike.

The scale arrived faster than the scrutiny. HiddenLayer counts 70,000+ skills on ClawHub, with malicious ones appearing in the wild within a couple of months of the platform's launch. The core problem is structural:

• Skills are not cryptographically signed.
• They are rarely vetted or reviewed before publication.
• Anyone with a GitHub account can publish one.
• Agents ingest and execute whatever is inside.

As the researchers put it, an enterprise that audits its software dependencies but places no controls on what context files its agents consume has a significant blind spot. Auditing that only hunts malicious code also misses skills that are technically clean but behaviourally compromised — a category that outcome-only testing will never catch, since benchmarks tell us what an agent achieved, not how or why, per recent evaluation research.

This is the gap SkillzBench is being built to close. The free, independent directory and MCP connector are designed around Verify (pin every version to a commit SHA, license-check, security-scan, quality-judge) and Benchmark (run each skill in a sandbox with and without the same model, then publish the Lift in percentage points). Published research found roughly 26% of skills carry at least one vulnerability — so before you hand an agent a skill, someone should have proven it helps.

Skills, proven. Join the SkillzBench waitlist for early access to the free directory and MCP connector.

How to Test a Skill Properly: Sandbox, With and Without, Measure the Lift

If you install a skill and the agent performs better, is that the skill working — or the model, the prompt, or luck? The only honest way to know is a controlled test: same model, same tasks, with and without the skill, and a number that isolates the difference.

That number is what evaluation researchers call a lift. The methodology matters because one case study found frontier agent capability was under-elicited by nearly 50% when tests tracked only final task success — pass/fail scores hide the actions, tool calls and reasoning that produced each outcome.

The playbook has four steps:

• Run it in a sandbox. Agents use tools across many turns and modify state, so mistakes propagate and compound; a controlled environment keeps those runs from touching your real files.
• Test with and without. The same model, the same tasks, twice — once bare, once with the skill loaded. The gap in percentage points is the skill's actual contribution.
• Watch the logs, not just the verdict. Benchmarks tell you what the agent achieved; only logs reveal how and why it got there.
• Grade with all three grader types. Anthropic's engineering guidance describes code-based, model-based and human graders as complementary — code checks correctness objectively, human graders catch nuance.

As you review logs, watch for the failure modes Databricks' evaluation guide catalogs: hallucinated tool calls, dead-end reasoning, tool overuse, and stale memory. A skill that fails by deleting a Git branch poses an entirely different risk than one that fails on a syntax error — yet outcome-only scores obscure that distinction completely.

This is precisely the approach SkillzBench is being built around. The free directory will benchmark every skill in a sandbox with the same model, with and without the skill, and publish the Lift in percentage points separately from its 0–100 trust score — so ad money and ratings never blur what a skill actually contributes.

The waitlist is open now. If you want to see skills proven before you trust them, join it and get early access to the benchmarks.

Social proof line: Published research found ~26% of skills carry at least one vulnerability, and security researchers warn skills are unsigned and rarely vetted. SkillzBench is being built to change that — verify, benchmark, and see the Lift before you trust a skill.

Your Next Step: Match the Model to the Task, Prove the Skill

So which model should you actually pick? The honest answer, as one hands-on comparison puts it, is "not which model is best, but which model is best for what" — and the benchmark data backs that up.

Start by matching the model to the task. Aggregated benchmark data shows each family leads a different domain: Claude dominates coding and long-horizon agentic work, GPT leads mathematics and PhD-level science reasoning, and Gemini leads knowledge breadth, multimodal and speed-sensitive tasks. A useful rule of thumb:

• Coding and agentic work: Claude leads SWE-bench Verified at 83.5% and Terminal-Bench 2.0 at 90.2%, with METR time horizons far ahead of rivals.
• Math and science: GPT leads GPQA Diamond at 94.6% and scores a perfect 100% on OTIS Mock AIME.
• Breadth, speed and cost: Gemini leads Humanity's Last Exam at 46.44% and offers high-output models at low cost — one Flash-Lite variant runs 449 tokens/s at $0.19 per million tokens.

But picking the right model is only half the job. The other half is proving the skill actually helps before you trust it. Recent evaluation research found that outcome-only benchmarks are unreliable — in one case study, frontier agent capability was under-elicited by nearly 50%, and logs revealed that "benchmarks tell us what the agent achieved; only logs reveal how and why."

That verification gap matters even more for agent skills. Security researchers warn that skills are not cryptographically signed and are rarely vetted, with malicious skills appearing in the wild within months of the ecosystem's launch — and published research found roughly 26% of skills carry at least one vulnerability. A skill that looks helpful may quietly compromise behavior.

This is exactly what SkillzBench is being built to fix: an independent directory that pins every skill version to a commit SHA, security-scans and quality-judges it, then benchmarks each skill in a sandbox with the same model — with and without the skill — so you can see the Lift in percentage points before you deploy it. A free MCP connector will let any agent load only the skills you choose, with alerts when upstream versions change and need re-testing.

Everything stays free — the directory, the MCP connector, benchmarks and alerts — supported by advertising that never touches scores or listings. Join the waitlist at skillzbench.com to get early access when it launches, and stop guessing whether a skill actually works.