Tool Calling Explained: From Text Generator to Agent

Ask a large language model about the weather in Tokyo without any tools, and it will produce a confident, fluent answer that may be completely made up. That is the gap tool calling exists to close. Tool calling, also called function calling, is the ability of AI models to interact with external tools, APIs, or systems rather than relying solely on what they learned during training, according to IBM.

The mechanism follows a consistent loop across every major provider. The model receives your prompt along with a set of tool definitions, decides whether one is needed, and emits a structured call — say, get_weather with location: Tokyo — formatted as JSON. Your application code then executes that call against the real API or database, returns the result to the model, and the model composes its final answer from verified data. OpenAI's function calling documentation describes exactly this multi-step exchange, and IBM breaks it into six stages, from recognizing the need for a tool to refining the result.

This loop is what separates an agent from a chatbot. As one analysis puts it, function calling is the mechanism that turns a language model from a text generator into a tool-using agent — capable of querying databases, updating records, and triggering workflows instead of producing plausible-sounding approximations.

But the decision to call a tool is itself a failure point. Research from MPI-SWS and Ruhr University Bochum found that models frequently misjudge whether a call is necessary, useful, or affordable — tool calling "is not universally beneficial" and can sometimes hurt performance. Docker's evaluation of 21 models across 3,570 test cases found tool-selection F1 scores ranging from 0.974 for GPT-4 down to 0.484 for a purpose-built tool-calling model, along with failure modes like calling tools in response to a simple greeting.

The practical lessons for anyone building agents:

• Whether to call a tool matters as much as how — models' self-assessments of need often diverge from reality.
• Empirical, same-model testing is the only reliable way to know which tools actually help.
• Tool overload has real costs — Anthropic reports agents with thousands of tool definitions can burn hundreds of thousands of tokens before even reading a request.
• Provider implementations differ in small but bug-prone ways, such as OpenAI returning arguments as a JSON string while Anthropic returns parsed objects.

This is exactly why SkillzBench is building its Benchmark pillar: testing agent skills in a sandbox with the same model, with and without the skill, and publishing the Lift in percentage points. Because when a model can act instead of just talk, "probably helps" is no longer good enough — skills, proven.

The Hidden Problem: Models Misjudge When to Call Tools

Most teams assume a model that can call tools will call them wisely. The evidence says otherwise: the hardest part of tool calling isn't executing the call — it's deciding whether the call should happen at all.

Academic researchers from MPI-SWS and Ruhr University Bochum evaluated tool calls along three dimensions — necessity, utility, and affordability — and found that models' self-assessed need and utility are misaligned with reality. Their conclusion is blunt: tool calling is not universally beneficial, and in some cases can even hurt performance. Models also struggle to make rational decisions under affordability constraints, meaning they'll happily burn budget on calls that add nothing.

Docker's independent evaluation makes the capability gap concrete. Testing 21 models across 3,570 test cases, they measured tool selection F1 scores ranging from 0.484 to 0.974 — a spread wide enough to make or break a production agent. Purpose-built tool-calling models underperformed expectations, contradicting the assumption that specialization guarantees quality. Recurring failure modes included:

• Eager invocation — calling tools in response to simple greetings
• Wrong tool selection when multiple tools are available
• Invalid or malformed arguments sent to valid tools
• Ignored tool responses, wasting the call entirely

Context overload compounds the problem. Anthropic's engineering team found that agents with thousands of tools may process hundreds of thousands of tokens before even reading a request, and in one example, presenting MCP servers as code APIs cut token usage from 150,000 tokens to just 2,000 — a 98.7% time and cost saving. More tools in context doesn't just cost money; it measurably degrades the agent.

The uncomfortable takeaway: the problem isn't capability — it's judgment. A model needs to know when to call, which tool to call, and whether calling helps at all. Those are three separate decisions, and models routinely get them wrong in ways that vendor demos never show.

This is exactly why empirical, side-by-side testing matters. SkillzBench is being built around that principle: benchmarking every skill in a sandbox with the same model, with and without the skill, to measure the actual Lift rather than trusting that more tools means better outcomes. Because as the research shows, sometimes the tool call is the mistake — and the only way to know is to test it.

MCP and the Converging Ecosystem: One Standard, New Trust Questions

When Anthropic introduced MCP in November 2024, few predicted how quickly the rest of the industry would fall in line. Today, the protocol has been adopted by OpenAI and Google, and OpenAI is sunsetting its own Assistants API in favor of MCP by mid-2026. What started as one company's integration layer is now the de-facto standard for connecting agents to tools and data, with thousands of pre-built MCP servers for services like GitHub, Slack, and databases.

Standardization solves real problems. Teams no longer rewrite integrations for every provider, and the "USB-C port" analogy is apt: one connector, many devices. But convergence cuts both ways. When every agent can reach every tool through a shared standard, a single vulnerable server or poorly scoped permission becomes a risk multiplied across the entire ecosystem rather than isolated to one app.

The security concerns are well documented. Practitioners now treat prompt injection, least-privilege access, schema validation, and allowlists as baseline requirements for any tool-calling deployment — including indirect injection, where malicious instructions arrive inside tool outputs rather than user prompts. And the risk surface keeps growing: published research suggests roughly 26% of agent skills carry at least one vulnerability, meaning about one in four shared skills a team might install comes with an unknown defect attached.

This is where curation matters more than volume. SkillzBench, currently in development by AIQ Labs, is building around exactly this gap: continuously crawling and deduplicating skills from GitHub, pinning every version to a commit SHA, and running security scans, license checks, and sandboxed benchmarks before anything earns a trust score. Its planned delivery model — a single MCP server with per-agent loadouts — reflects a least-privilege principle in practice: each agent sees only the skills and tools explicitly chosen for it.

Three questions worth asking before adopting any shared tool or skill:

• Does it actually improve agent performance, or just add context weight? Anthropic's own engineering work shows tool overload is real — agents with thousands of tools can burn hundreds of thousands of tokens before even reading a request.
• Who maintains it, and is the version pinned? Unpinned dependencies mean upstream changes can silently alter behavior.
• Has it been tested with your model, in your workflow? Docker's 21-model evaluation found tool-calling quality varies dramatically, with F1 scores ranging from 0.484 to 0.974 — specialization is no guarantee of quality.

The ecosystem has converged on one standard. The next challenge is trust: knowing which of the thousands of available skills genuinely lift agent performance — and which belong nowhere near your agents.

Prove It, Don't Assume It: Benchmarking Tools and Skills the Right Way

Your agent just gained a new skill. Is it actually better now — or just noisier? The honest answer is that most teams have no idea, because they've never measured it.

The research is blunt about why this matters. A peer-adjacent study from MPI-SWS and Ruhr University Bochum found that models routinely misjudge whether a tool call is necessary or useful, and that "tool-calling is not universally beneficial — it does not always improve utility, and in some cases can even hurt." In other words, the model can't tell you whether a tool helps. You have to test it.

The proof comes from systematic testing, not vendor claims. Docker's independent evaluation of 21 models across 3,570 test cases revealed tool-selection F1 scores ranging from 0.974 (GPT-4) down to 0.484 — and purpose-built tool-calling models that underperformed general ones. As Docker put it, systematic testing turned "we don't know which model to pick" into "three great options, each with clear pros and cons."

The same discipline applies to skills and tools. The playbook looks like this:

• Benchmark with and without — run the same model in a controlled sandbox, once with the skill enabled, once without, and measure the Lift in percentage points. Anecdotes don't count; deltas do.
• Curate per-agent loadouts — agents see only the tools and skills you've chosen for them. Anthropic's engineering team reports that agents with thousands of tools may process hundreds of thousands of tokens before reading a request, so every tool you expose is a cost you pay.
• Pin versions and scan before enabling — published research found roughly 26% of skills carry at least one vulnerability, so every skill should be pinned to a commit SHA, license-checked, and security-scanned before it touches your agent.

This is exactly the workflow SkillzBench is building for agent skills: Discover, Verify, Benchmark, Equip — continuously crawling skills from GitHub, validating and scanning each version, measuring real Lift in a same-model sandbox, and delivering curated per-agent loadouts through a single MCP server. And because skills change upstream, the loop repeats: track, re-scan, re-benchmark, notify.

The principle underneath all of it is simple: measure the delta, not the hype. A skill that adds 12 points on your task is worth equipping. One that adds nothing — or quietly injects risk — isn't. Until the ecosystem makes that measurement routine, most agents will keep running on assumptions.

If you want that measurement built in from day one, join the SkillzBench waitlist at skillzbench.com — skills, proven.