How Do AI Models Work?

The Black Box Problem: Why 'The Model' Is Only Half the Story

Ask an AI the same question twice and you can get two different answers — and most explanations you'll find online stop at "it predicts the next token." That's technically true and practically useless. It doesn't tell you why responses are slow, why costs spike, or why your AI agent nails a task on Monday and fumbles it on Tuesday.

Here's the reframe that changes everything: a model is trained once, then run forever. Training adjusts billions of parameters over weeks or months on GPU clusters; inference — the part you actually interact with — is a read-only, forward-only process that generates output one token at a time, without updating anything the model learned. According to IBM's technical explainer, LLM inference is precisely this: running a pre-trained model to generate output tokens for new prompts, with no parameter updates.

That split matters because of where the money goes. Inference is estimated to account for 80–90% of a model's total lifecycle compute costs in production, with training taking just 10–20%, according to industry analysis from SambaNova. If you're adopting AI at work, your costs and quality issues live almost entirely on the inference side — which is exactly the half most explanations ignore.

So what actually happens when you hit enter? The pipeline has four well-defined stages:

• Tokenization: your prompt is split into tokens from a vocabulary that typically ranges from 32,000 to 128,000 entries.
• Prefill: the model processes your entire input in parallel and builds a memory structure called the KV cache — this phase is compute-bound.
• Decoding: output is generated autoregressively, one token at a time, and each new token requires reading the full model weights from memory — this phase is memory-bandwidth-bound, which is why LLM responses are inherently slower than a database query.
• Detokenization: tokens are converted back into human-readable text.

Once you see this, the "black box" starts making sense. Attention computation grows quadratically with sequence length, so long prompts don't just cost more — they cost disproportionately more. A KV cache for a 70-billion-parameter model with a 4,096-token context can consume several gigabytes of GPU memory. And because the model reads its weights from memory for every single token, the quality, size, and organization of what you feed it — the context — directly shapes speed, cost, and reliability.

This is why experts keep insisting the issue is rarely the model itself; it's whether the agent has access to the right context at the right time. In agentic systems, context quality — not model capability — is the reliability bottleneck. That's the lever you actually control: better instructions, curated tools, and cleaner inputs. It's also why the team at SkillzBench is building a free directory of verified agent skills with per-agent loadouts — because when inference dominates cost, controlling what your agent sees at inference time is how you control quality, speed, and spend.

What Happens When You Hit Enter: The Four Stages of Inference

The moment you hit enter, a surprisingly mechanical sequence begins — no magic, just four well-defined stages running in milliseconds. Understanding them explains why a chatbot's answer takes seconds while a database query takes milliseconds, and why your prompt length matters more than you might think.

Stage one: tokenization. Before the model sees anything, your text is chopped into tokens — chunks of words or subwords — matched against a vocabulary that typically ranges from 32,000 to 128,000 tokens. The model never reads letters or sentences; it reads these token IDs as numbers.

Stage two: prefill. The model processes your entire prompt in parallel, running it through dozens of transformer layers (a 70B-parameter model like Llama 3 has 80 layers and 64 attention heads). This pass also builds the KV cache — a stored snapshot of the prompt's internal representations — so the model never has to recompute it. For a 70B model with a 4,096-token context, that cache alone can consume several gigabytes of GPU memory. Prefill is compute-bound: it's the fast, parallel part.

Stage three: decoding. Here's the bottleneck. The model generates one token at a time, each new token depending on all the previous ones. Because every single token requires reading the full model weights from memory — roughly 140 GB at FP16 precision for a 70B model — decoding is memory-bandwidth-bound, not compute-bound. This is why an LLM response is fundamentally slower than a database lookup: the database reads an index; the model streams its entire brain from memory for every word it writes.

Stage four: detokenization. The output token IDs are mapped back to human-readable text and streamed to your screen.

Two practical consequences follow from this architecture:

• Context windows are quadratic in cost. Attention computation grows quadratically with sequence length, so doubling your prompt more than doubles the work — long, rambling prompts cost disproportionately more compute.
• Speed is psychological. A 200ms time-to-first-token "feels responsive" even if total generation takes seconds, which is why responses stream word by word rather than appearing all at once.
• Every token is billed compute. Inference accounts for an estimated 80–90% of a model's total lifecycle compute costs, so verbose prompts and verbose agents carry real, compounding expense.

That last point matters most for anyone building agents. A multi-agent pipeline can burn roughly 15x the tokens of a single chat interaction, and much of that cost is context stuffed into the prompt — including instructions and skills the agent doesn't actually need. Since agent reliability depends on context quality rather than raw model capability, curating exactly what enters that expensive prefill stage is one of the highest-leverage optimizations available. That's the principle behind SkillzBench's per-agent loadouts: agents only see the skills you've chosen, keeping the pipeline lean. The directory and free MCP connector are being built to make that curation — verified, benchmarked, and security-scanned — straightforward for any agent stack.

The New Scaling Law: Buying Accuracy at Inference Time

For a decade, the path to smarter AI was simple: train a bigger model. That era is ending. As pre-training scaling shows diminishing returns, researchers have identified a third lever — giving models more compute at the moment they answer, an approach known as test-time compute.

The idea reframes accuracy as something you can buy at inference time, not just at training time. The clearest demonstration came at NVIDIA's GTC conference, where DeepSeek R1 answered a question using 20 times more tokens and 150 times more compute than a traditional model, according to industry analysis. The model "thinks" before it answers, burning reasoning tokens the way a human drafts and revises.

Three techniques define this new scaling law:

• Reasoning tokens — chain-of-thought "thinking" that trades latency and cost for accuracy on hard problems.
• Retrieval-Augmented Generation (RAG) — pulling relevant facts into the context window before answering, improving reliability without retraining.
• Adaptive computation — deciding per query whether a cheap, fast pass suffices or multi-step reasoning is worth the bill.

The economics behind this shift are dramatic. The cost of inference per million tokens collapsed from $20 to $0.07, per the Stanford 2025 AI Index — a price drop that makes "spend more compute to think longer" viable for everyday workloads. Inference now represents an estimated 80–90% of a model's total lifecycle compute costs, which is precisely why optimizing it matters more than training ever did.

Compression is the other cost lever. Quantization — storing model weights at lower numerical precision — cuts memory requirements by 2–4x, and INT8 output is nearly indistinguishable from full-precision FP16 for most tasks, according to technical benchmarks. INT4 shows measurable quality loss but runs on consumer hardware. For teams running models locally, that tradeoff often decides what's feasible.

These levers converge on a practical business decision: routing. The market is bifurcating into frontier cloud models and smaller, highly distilled specialist models, and routing simple tasks to smaller models is now a mainstream cost strategy. Walmart's AdaptJobRec system demonstrated the payoff of selective reasoning — applying agentic computation only where needed — cutting response latency by up to 53.3% while improving recommendation quality.

The same restraint principle applies to agents. A multi-agent pipeline can burn roughly 15x the tokens of a single chat interaction, so deciding which tasks deserve heavy reasoning — and which deserve a small model and a tight context — is where budgets are won or lost. Context quality, not raw model capability, drives agent reliability, and equipping agents with only the verified skills and tools they actually need keeps both cost and risk contained. SkillzBench is building exactly that discipline: a free directory of agent skills with per-agent loadouts, so each agent sees only the context you've chosen — and nothing it shouldn't.

Context Beats Capability: Why Agents Fail and How to Fix Them

Here's a counterintuitive truth: when AI agents fail, the problem is usually not the model. It's the context.

Neo4j's field CTO Jesús Barrasa puts it plainly in a series of production agent case studies: "The issue is rarely the model itself — it's whether the agent has access to the right context at the right time." Across deployments, the same failure patterns keep appearing, and none of them trace back to raw model capability:

• Hallucinations caused by incomplete domain knowledge
• Lost state as long workflows accumulate errors
• Brittle prompts that break on edge cases
• Tool misuse when the agent lacks clear instructions

This matters because agents are expensive. According to industry analysis, a multi-agent pipeline can burn roughly 15 times as many tokens as a single chat interaction — and enterprises report burning through annual AI budgets in months once agents become genuinely useful. Every wasted token spent re-deriving context the agent should have had from the start is money out the door.

The fix is what practitioners call context engineering: giving the agent exactly the right instructions, knowledge, and tools at each step — no more, no less. Neo4j's conclusion from real deployments is that effective agent systems require "orchestration and restraint, not maximum autonomy." Walmart proved the point by applying agentic reasoning selectively, cutting response latency by up to 53.3% while improving recommendation quality.

Agent skills are one of the most practical context-quality levers available. A skill is a packaged set of instructions — like a SKILL.md file — that teaches an agent how to perform a specific task well, replacing vague prompts with tested procedures. But skills cut both ways: published research found that roughly 26% of published skills carry at least one vulnerability, and installing one blindly means handing an untrusted instruction package to an agent that touches real data and real workflows. As NetBox Labs' CTO Shannon Weyrick warns, "you can't secure or manage what you can't see."

That's the gap SkillzBench is building to close: a free directory of agent skills that are security-scanned, version-pinned, and benchmarked with and without the skill, so you can see the measured Lift before equipping anything. Its per-agent loadouts embody the restraint principle directly — the agent only sees the skills you chose for it.

The lesson from the inference pipeline is the same at every layer: the model is a read-only engine. Reliability comes from the architecture around it — the context you supply, the tools you allow, and the boundaries you enforce.

Putting It to Work: Equipping Your Agents with Verified Context

Here's the uncomfortable truth about agents: the model is rarely the problem. Neo4j's field CTO Jesús Barrasa puts it plainly — "the issue is rarely the model itself — it's whether the agent has access to the right context at the right time." You now understand how inference works, so you're equipped to control the one variable that actually drives reliability: what your agent sees.

Start with an audit. Before adding anything, find out what context your agents currently receive — which instructions, tools, and skills are loaded into their context window on every run. This matters because attention costs grow quadratically with sequence length, and a multi-agent pipeline can burn roughly 15x the tokens of a single chat interaction. Bloated, irrelevant context isn't just slow; it's expensive and error-prone.

Next, apply selective loadouts. Effective agent systems require "orchestration and restraint, not maximum autonomy" — the same principle behind Walmart's decision to apply agentic reasoning selectively, which cut response latency by up to 53.3% while improving quality. Give each agent only the skills it needs, and nothing more.

Then demand verification for every skill you install. Published research found that roughly 26% of agent skills carry at least one vulnerability — an unmanaged inference-side risk when agents touch real data and real workflows. Before trusting any skill, check that it offers:

• Version pinning to a specific commit, so you know exactly what's running
• Security scanning and license validation, not just a README
• A transparent trust score built from security, quality, and maintenance signals
• Benchmark results that isolate the skill's actual impact

Finally, benchmark with-versus-without. The same "test-time compute" logic that makes reasoning models more accurate applies to skills: measure whether a skill genuinely improves output before deploying it. That's the gap SkillzBench is building to close — a free, independent directory of agent skills, a free MCP connector that lets you toggle skills per agent so each one sees only what you chose, and sandboxed benchmarks that publish each skill's Lift in percentage points, tested with and without the skill on the same model. Every skill is pinned to a commit SHA, security-scanned, and scored on a 0–100 trust scale, with Lift always shown separately and money never touching the scores.

Everything — directory, connector, benchmarks, alerts — is free, and it's being built now. If you want to move from understanding inference to controlling it, join the SkillzBench waitlist and be first in line when it launches.