How LLM works diagram?

Why Everyone Struggles to Picture How an LLM Actually Works

You've typed a message to ChatGPT and gotten a reply that feels, unsettlingly, human. Yet almost nobody — including many people who build with these models daily — can sketch what actually happens between your keystroke and that answer.

The best starting point is surprisingly humble. As Stanford HAI puts it, "GPT-3 is trained to predict the next word in a sentence, much like how a text message autocomplete feature works." That analogy demystifies a 175-billion-parameter system trained on 570 gigabytes of text — but it's also where most explanations stall.

Here's the problem: the explanations you'll find online live at two unhelpful extremes.

• Too simplistic: popular plain-language guides deliberately skip transformers and attention entirely — one widely shared explainer openly admits it is "intentionally very simplistic," leaving readers with no idea why models handle long-range context at all.
• Too technical: the accurate sources open with formulas like Attention(Q,K,V) = softmax(Q×Kᵀ/√dₖ)×V, which is mathematically correct and completely impenetrable to most professionals.
• Too fragmented: inference details — prefill, decoding, the KV cache — live in separate vendor docs, so no single picture ties the pipeline together.

The gap matters because the omitted middle is where the interesting stuff happens. Attention is the mechanism that lets a model connect a pronoun at the end of a paragraph to a name at the beginning — something peer-reviewed research confirms earlier architectures like recurrent networks captured far less effectively. And the 2017 "Attention Is All You Need" paper that introduced all of this has been cited more than 250,000 times, ranking among the top ten most-cited papers of the 21st century — so the core idea is clearly learnable, just rarely taught visually.

There's also a practical reason to want the full picture. Once you understand that an LLM is fundamentally a tokenizer plus a neural network whose output gets appended back to its input in a loop, the role of everything built on top — memory, tool access, agent skills — snaps into focus. LLMs gain those capabilities only through a software harness, which is exactly where modern agent toolkits live. SkillzBench is building a free directory and MCP connector for exactly those harness extensions, so understanding the base diagram makes the whole agent stack easier to reason about.

What's needed is one master diagram that starts with the autocomplete intuition and layers in the real machinery — tokenization, attention, next-token prediction, and the loop — without requiring a linear algebra background. That's what this article delivers.

The Master Diagram: From Your Text to the Model's Next Word

Every time you type a prompt, an intricate pipeline fires: your words become numbers, numbers flow through layers of attention, and out pops exactly one token. Then the whole thing runs again. That loop, repeated hundreds of times per response, is the master diagram of how an LLM works.

The journey starts with tokenization. Your raw text is split into tokens — words or word fragments — and mapped to numbers the model can process, the essential first stage of every inference pipeline according to IBM's technical explainer. Each token then receives an embedding (a vector of meaning) plus a positional encoding, since transformers process tokens all at once and need order injected explicitly.

Next come the transformer layers. Here, self-attention lets every token look at every other token simultaneously, regardless of distance. The 2017 paper that introduced this architecture, "Attention Is All You Need," has been cited more than 250,000 times as of 2026, making it one of the top ten most-cited papers of the 21st century, per its Wikipedia entry.

Why did transformers win? Two reasons:

• Parallelizability — unlike RNNs, transformers process whole sequences at once, enabling GPU-accelerated training of far larger models.
• Long-range dependencies — attention captures contextual relationships across distant tokens far more effectively than recurrent networks, per peer-reviewed analysis in Nature.
• Scalability — the original transformer had just 100M parameters; GPT-3 scaled to 175 billion, trained on 570 GB of text.

Finally, the model predicts the next token. The highest-value output neuron — one per vocabulary word — becomes its guess. That token is then appended back to the input and fed through the network again, which is precisely why LLMs output one word at a time, as one plain-language breakdown puts it. Stanford HAI frames the whole system simply: it's trained to predict the next word, much like text message autocomplete.

Inference itself splits into two phases: a compute-heavy prefill that processes your full prompt in parallel, and a memory-bound decode phase that generates tokens one by one. A KV cache stores intermediate attention results so earlier tokens aren't recomputed — though attention cost still grows quadratically with sequence length.

This loop also explains the "software harness" opportunity: since LLMs alone have no memory or tools, agent skills extend them with instructions and capabilities. SkillzBench is building a free directory and MCP connector so you can discover, verify, and benchmark those skills — join the waitlist to see which ones actually deliver measurable lift.

Under the Hood: Prefill, Decode, and Why Your Chatbot Forgets

Ever watched your chatbot confidently "remember" your first question, then go blank ten turns later? The answer lives in how inference actually runs — and it's a tale of two phases.

When you hit send, the model first processes your entire prompt in parallel. This is prefill: compute-intensive, GPU-saturating work that initializes the KV cache — a store of intermediate attention results so earlier tokens never need recomputing. According to IBM's technical explainer, prefill is where the heavy math happens.

Then comes decode: the model emits exactly one token at a time, each depending on all previous tokens, appending its output back to the input and re-running. Decode is memory-bound rather than compute-bound — the bottleneck is moving data, not multiplying numbers. SambaNova notes that decode "sets the cost and speed of most inference today," a claim detailed in their inference breakdown.

• Prefill: parallel, compute-bound, builds the KV cache
• Decode: sequential, memory-bound, one token per pass
• KV cache: prevents recomputing attention for earlier tokens
• Inference is read-only — model weights never update during generation

Here's the catch: attention cost grows quadratically with sequence length. Double your context and you quadruple the attention computation. That's the price of the self-attention mechanism from the 2017 "Attention Is All You Need" paper, which replaced recurrence with scaled dot-product attention and has since been cited over 250,000 times, per Wikipedia's coverage.

The context window is the honest answer. As one plain-language explainer puts it, the window is proportional to the number of input neurons — a fixed, finite count. GPT-2's small version had just a 1k token window, per Wikipedia. Once your conversation exceeds that budget, the oldest tokens simply fall off the edge. The model isn't being rude; it physically cannot see them.

The fix for real workflows is a software harness — external memory and tool access layered around the model. Agent skills are exactly that, and SkillzBench is building a free directory and MCP connector to discover and benchmark them: skills, proven.

From Prediction to Action: The Software Harness and Agent Skills

A raw LLM does exactly one thing: predict the next token. It cannot remember yesterday's conversation, browse the web, or call your calendar API. Everything that looks like intelligence beyond prediction comes from the code wrapped around the model.

As the standard reference on LLMs explains, a software harness is what gives a model memory, tool access, and web search. The neural network itself just loops — token in, token out, appended back to the input, over and over. The harness is the layer that decides what to do with those tokens.

This distinction matters for the diagram, too. When you draw an LLM, the "brain" is only the prediction loop. Everything else — retrieval, context injection, tool calls — lives in harness code outside the model weights.

That harness layer is where agent skills come in. Skills are instruction packages (typically SKILL.md files) that tell coding agents like Claude Code, Codex, or Cursor how to handle a task. They are, in effect, pluggable harness extensions — and they are multiplying fast across GitHub.

Here is the catch: nobody can currently tell which skills actually help and which are dangerous. Published research found that roughly 26% of skills carry at least one vulnerability. A skill that "just" contains instructions can still steer an agent toward leaking data or running unsafe commands.

This is why verification and measurement matter as much as discovery:

• Security scanning of every skill version, pinned to a specific commit
• Benchmarking in a sandbox — the same model, run with and without the skill, to measure real Lift
• License and quality checks, so a skill is safe and legal to use
• Ongoing tracking of upstream changes, so an update that breaks or endangers a skill gets caught

SkillzBench is being built to do exactly this: a free, independent directory of agent skills, plus a free MCP connector that links any agent to the skills you choose. Every skill gets a trust score built from security, quality, license, maintenance, and community signals — and Lift is always reported separately, so a skill's usefulness is never confused with its safety.

If you are wiring tools and memory around a model, the harness is where your risk lives. Treating skills like the software they are — versioned, scanned, and measured — is the difference between an agent that works and one that quietly fails.

Put the Diagram to Work: Reading AI Claims with New Eyes

Once you can sketch the pipeline on a napkin — text in, tokens, attention, one word out, loop — every AI vendor pitch starts reading differently. The diagram stops being trivia and becomes a practical evaluation tool.

Decode sets the price tag. When a vendor claims blazing-fast inference, remember that inference runs in two phases: prefill processes your whole prompt in parallel, while decode generates output one token at a time. As SambaNova's technical explainer notes, decode is memory-bound and "sets the cost and speed of most inference today." A model that writes long answers pays for every token it emits.

Hallucination is a feature of the mechanism, not a bug. Because the model is fundamentally predicting the next plausible token — Stanford HAI describes it as "much like how a text message autocomplete feature works" — it can produce superficially plausible content that is factually inaccurate, as Stanford researchers have documented. Treat AI output as a first draft, never a final answer.

A few practical rules fall straight out of the diagram:

• Long prompts cost more: attention computation grows quadratically with sequence length, per IBM's inference analysis.
• "Forgetting" is structural — the context window is finite, so earlier conversation drops out.
• Reasoning models spend "thinking tokens" before answering, improving accuracy without retraining — but adding cost.
• Tool access and memory come from a software harness around the model, not the model itself.

That last point matters most for anyone building with agents. The raw LLM is just the prediction loop; everything useful — web search, memory, specialized instructions — gets bolted on. Agent skills are exactly that kind of harness extension, and choosing good ones is where real risk lives.

Finally, calibrate your expectations with the timeline. The transformer arrived in the 2017 paper "Attention Is All You Need," now cited over 250,000 times per Wikipedia. From there: GPT-3's 175 billion parameters in 2020, ChatGPT in late 2022, and reasoning models by 2024. Seven years from research paper to reasoning — plan accordingly.

If you're equipping agents with skills, SkillzBench is building a free, verified directory where every skill gets security-scanned and benchmarked for the lift it actually delivers. Join the waitlist at skillzbench.com to browse and test free agent skills before anyone else.