Same Model, Different Results: Why Agent Setup Matters More Than the LLM

Introduction

Two teams run the same model. One ships; the other debugs for weeks. The difference isn't the LLM — it's everything wrapped around it.

When Google Research evaluated 180 agent configurations across three model families, the results were striking: centralized coordination delivered +80.9% gains on parallelizable tasks, while every multi-agent variant degraded performance by 39% to 70% on sequential tasks — with the exact same underlying models (Google Research's controlled study). The model never changed. The setup did, and the setup decided the outcome.

This finding is converging from every direction. An arXiv position paper argues that benchmark scores reflect the entire agent system — harness, environment, evaluator, inference budget — not just the model, and that small stack differences can flip model rankings entirely. OpenAI's own deployment guide reaches the same conclusion from the field: tool clarity, not tool count or model choice, drives agent success (OpenAI's practical guide).

The practical implications cut across everything teams configure:

• Architecture — multi-agent setups amplified errors 17.2x in Google's study, versus 4.4x for centralized systems
• Tools and instructions — some implementations manage 15+ well-defined tools while others fail with fewer than 10 overlapping ones
• Harness design — the same model given tools and an iterative loop outperformed its prompt-only self in practitioner testing (one documented case study)

For teams using agent skills — the SKILL.md instruction packages now spreading across Claude Code, Codex, Cursor, and Gemini CLI — this raises an uncomfortable question. If setup dominates outcomes, and skills are setup, then which skills actually help? As one researcher put it, the markdown in a SKILL.md file isn't documentation; it's code running on a non-deterministic interpreter. And published research has found roughly 26% of skills carry at least one vulnerability.

That's the problem SkillzBench is being built to answer: benchmarking every skill in a sandbox with the same model, with and without the skill, so the Lift — the measurable difference — is visible before anything touches your agent.

The rest of this article breaks down what the research actually shows about where agent performance comes from, and why the question worth asking is no longer "which model?" but "which setup?"

Key Concepts

Your agent's benchmark score might say less about your LLM than about the scaffolding wrapped around it. That's the uncomfortable conclusion emerging from controlled research, and it reframes how teams should think about agent performance.

The strongest evidence comes from Google Research's controlled study of 180 agent configurations across three model families. The same underlying models swung from +80.9% gains with centralized coordination on parallelizable tasks to -39% to -70% degradation with multi-agent setups on sequential tasks. The model never changed — the setup did.

An arXiv position paper explains why: benchmark scores are shaped by the agent harness, environment, evaluator, and inference budget, not just the model. Two models scoring 62 and 58 on the same benchmark only support a model comparison under a shared execution setup. Small stack differences — inference engine, tool-call schema support, memory serialization policy — can flip model rankings entirely.

Production experience tells the same story. Neo4j's case-study roundup finds the issue is "rarely the model itself" — agents fail because they reason with insufficient or poorly organized context. And OpenAI's deployment guide reports some implementations managing more than 15 well-defined tools successfully while others struggle with fewer than 10 overlapping ones. Tool clarity, not tool count or model choice, drives outcomes.

The practical levers that separate good setups from bad ones:

• Architecture matched to task structure — parallelizable tasks benefit from coordination; sequential tasks suffer from fragmented "cognitive budgets"
• Tool definitions that are distinct and non-overlapping, not merely numerous
• Instructions and skills treated as executable artifacts, not prose — as researcher Jimmy Lin puts it, "the markdown in a SKILL.md file is code"
• Context quality and state management, which production failures trace back to far more often than reasoning errors

This is why SkillzBench is building its benchmark methodology around a fixed model: testing every skill in a sandbox with and without the skill, then publishing the Lift in percentage points. It isolates exactly the setup variable the research says matters most — and keeps that measurement separate from trust ratings, so a skill's measured impact is never confused with its security or maintenance posture.

The takeaway is simple but demanding: if setup dominates outcomes, then setup deserves the same rigor you'd give model selection. Your model is a constant; your harness is the variable you control.

Best Practices

If the same model can swing from +80.9% to -70% purely on configuration, as Google Research found across 180 agent configurations, then your setup decisions deserve as much rigor as your model choice. Here's how to make those decisions systematically.

Start with one agent, then earn complexity. OpenAI's deployment guide is blunt: "often a single agent with tools is sufficient," and customers "typically achieve greater success with an incremental approach" (OpenAI's practical guide). Google's data explains why — multi-agent setups amplified errors 17.2x in independent systems versus 4.4x in centralized ones. Add agents only when the task is genuinely parallelizable.

Fix your setup before blaming the model. Neo4j's production case studies find failures are "rarely the model itself" but context and architecture failures. Before swapping models, audit the things around it:

• Tool clarity: more than 15 well-defined tools can work; fewer than 10 overlapping ones can fail — similarity, not count, is the issue.
• Context quality: iterate on what the agent sees, not just how you prompt it.
• Inference stack: engine choice, tool-call schemas, and memory policies can flip model rankings entirely.
• Instructions: treat them as executable artifacts, not prose — "the markdown in a SKILL.md file is code."

Benchmark the setup, not just the model. An arXiv position paper warns that a benchmark score "does not, by itself, specify which of these questions it answers" — model capability and system quality get conflated. Run controlled comparisons where the model stays fixed and only the setup variable changes. That's the methodology SkillzBench is building around: testing skills in a sandbox with the same model, with and without the skill, and publishing the Lift in percentage points, kept separate from trust ratings so you always know what a score actually measures.

Treat skills like dependencies. Since skills are effectively code running on a non-deterministic interpreter, apply software hygiene: pin versions, security-scan before adoption, and re-verify when upstream changes land. Published research has found roughly 26% of skills carry at least one vulnerability, so blind installation is a real risk.

Finally, keep a single source of truth for what each agent sees. Per-agent loadouts — where an agent only sees the skills and tools explicitly enabled for it — reduce the overlap and ambiguity that reliably degrade performance regardless of model intelligence.

Implementation

Knowing that setup drives outcomes is one thing; operationalizing it is another. The good news is that the research points to a concrete playbook any team can run this week.

Start by auditing your architecture against your task structure. Google Research's controlled study of 180 agent configurations found the same models swung from +80.9% gains to -39% degradation purely based on configuration. Before adding agents, ask whether your tasks are actually parallelizable — if they're sequential, a single well-equipped agent will likely outperform a committee.

Next, treat instructions and tools as code, not prose. As one practitioner observes, "the markdown in a SKILL.md file is code" running on a non-deterministic interpreter. That means versioning, review, and testing — the same discipline you'd apply to any dependency. OpenAI's deployment guide reinforces this: teams successfully manage more than 15 well-defined tools while others fail with fewer than 10 overlapping ones, so audit for clarity and redundancy, not raw count.

A practical implementation checklist:

• Pin every instruction package to a specific version so results are reproducible.
• Test changes in isolation — one variable at a time, with a fixed model, so you can attribute the Lift to the setup change itself.
• Evaluate against your actual stack rather than generic benchmarks, since small stack differences can flip model rankings entirely.
• Keep benchmark deltas separate from quality ratings — collapsing them conflates model, harness, and environment effects.

Finally, iterate on context before prompts. Production case studies consistently show that failures stem from missing or poorly organized context, not faulty reasoning — successful teams iterate on system behavior rather than endlessly tuning prompts.

This is exactly the workflow SkillzBench is being built to support: sandboxed benchmarking of skills with and without the skill on a fixed model, version-pinned verification, and per-agent loadouts that control exactly what each agent sees. If you want your agent's skills to be measured rather than guessed at, join the waitlist at skillzbench.com.

Conclusion

The same model can gain 80.9% or lose up to 70% depending purely on how you configure the system around it — that is the uncomfortable truth running through every study we've examined. When Google Research evaluated 180 agent configurations across three model families, the swing came from architecture, not model capability (Google Research's controlled study). If you're benchmarking models without controlling setup, you're benchmarking noise.

The practical takeaway is straightforward: stop asking which model is best and start asking which setup is best for your task. The evidence points to a short list of levers that consistently matter more than the model choice itself:

• Match architecture to task structure — parallelizable work benefits from coordination, while sequential tasks degrade badly under multi-agent overhead.
• Prioritize tool clarity over tool count — OpenAI reports implementations managing 15+ well-defined tools successfully while others fail with fewer than 10 overlapping ones (OpenAI's deployment guide).
• Treat instructions as code — a SKILL.md file runs on a non-deterministic interpreter, so it deserves versioning, validation, and security scrutiny like any executable artifact.
• Evaluate against your actual stack, not abstract benchmarks — small differences in harness, evaluator, or inference budget can flip model rankings entirely (as one position paper argues).

This is also why the "more agents" heuristic fails so often. Error amplification runs 17.2x in poorly coordinated multi-agent systems versus 4.4x in centralized ones, and production failures are, as Neo4j's case studies put it, rarely model failures — they're context and architecture failures. The model was never the problem.

That framing is what SkillzBench is being built around. Because setup dominates outcomes, the lab's benchmark methodology tests every skill in a sandbox with the same model — with and without the skill — and publishes the Lift in percentage points, always kept separate from the trust score. Version pinning, security scanning, and per-agent loadouts follow from the same logic: if the harness is the variable that matters, measure and control the harness.

The next time a benchmark tells you model A beats model B, ask what else differed. The honest answer is usually: almost everything.

Skills, proven. SkillzBench is in active pre-launch development — join the waitlist to be first in line when benchmarking, verification, and per-agent loadouts go live.

ctaText: Skills, proven. Join the SkillzBench waitlist for early access. socialProofText: Built by AIQ Labs' independent testing lab — benchmarking every skill with the same model, with and without it.