How LLMs Actually Work
A large language model is a neural network - almost always a Transformer - trained on trillions of tokens to predict the next token, then post-trained to follow instructions.
Last updated: August 2026
Your text goes in - the model will continue it one token at a time.
Large language models can feel like magic, but the core idea is refreshingly simple: an LLM is a neural network - almost always a Transformer - trained to predict the next token. Feed it text and it turns each piece into a number called a token, passes those numbers through many layers that weigh how words relate to one another (a mechanism called attention), and outputs a probability for every possible next token. It picks one, appends it, and repeats - generating your answer a token at a time. That single trick, scaled up, is behind everything an LLM does: answering questions, writing code, summarizing, translating. Frontier models are first pretrained on 10-30 trillion tokens of text and code to learn broad patterns of language, then post-trained (instruction tuning and reinforcement learning from feedback) to become helpful, steerable assistants that follow instructions. Understanding this pipeline demystifies the model's strengths and its failure modes: it is astonishingly fluent because it has seen so much language, and it can 'hallucinate' confident nonsense because it is predicting a likely continuation, not looking up a verified fact.
“An LLM doesn't look up answers. It predicts the most likely next token, one at a time - which is why prompts and context matter so much.”
| Component | What it does | Example |
|---|---|---|
Tokens | Text is split into small units (tokens) and converted to numbers the model can process. | 'unbelievable' -> 'un', 'believ', 'able'. |
Embeddings | Each token becomes a vector that captures meaning, so similar words sit near each other. | 'king' and 'queen' land in nearby regions. |
Transformer | The architecture (from the 2017 'Attention Is All You Need' paper) that processes all tokens together in layers. | Every modern LLM is a Transformer. |
Attention | Lets each token weigh how strongly every earlier token influences it - capturing context. | 'it' correctly linked back to 'the cat'. |
Context window | The maximum span of tokens the model can consider at once - its working memory. | Everything you paste must fit the window. |
Next-token prediction | The model outputs a probability for every possible next token and samples one. | 'The sky is ___' -> 'blue' (high probability). |
Sampling / temperature | Controls how deterministic vs varied the choice of next token is. | Low temp = focused; high temp = creative. |
Pretraining | Learning broad language and code patterns from a web-scale corpus (10-30T tokens). | The 'base model' before it can chat. |
Post-training | Instruction tuning and reinforcement that make the base model helpful and steerable. | Turns a text predictor into an assistant. |
What this means in practice
- 1It predicts, it doesn't 'know'. - Answers are likely continuations, not database lookups - which is why models can sound confident yet be wrong (hallucinate).
- 2Context is everything. - The model only sees what's in its context window; better prompts and retrieved context yield better predictions.
- 3Sampling controls creativity. - Temperature and related settings trade off deterministic vs varied outputs from the same probabilities.
- 4Recency and framing matter. - How you phrase and order a prompt shifts the probabilities - small wording changes can change the answer.
- 5Bigger training helps, but isn't magic. - More data and parameters improve capability, but grounding (RAG) and good prompts still drive accuracy.
- 6Knowledge has a cutoff. - A model only knows what was in its training data unless you supply fresh context or tools.
Be specific
State the task, audience, and constraints precisely instead of leaving them implicit.
Prompt
'Summarize for a non-technical exec in 3 bullets.'
Give context
Paste the relevant material so the answer is grounded in your data, not the model's memory.
Prompt
Include the doc you want summarized.
Show an example
Provide one or two examples of the input/output you want (few-shot).
Prompt
Show a sample summary in your preferred style.
Specify the output format
Ask for a structure - a list, table, or JSON - so results are predictable.
Prompt
'Return a markdown table with 3 columns.'
Iterate
Refine the prompt based on what went wrong, adding rules or examples.
Prompt
'Shorter, and cite the source line.'
Common misconceptions
- 1'The model looks things up.' - It doesn't - it predicts likely text. Facts are only reliable when grounded via context or tools.
- 2'A confident answer is a correct answer.' - Fluency and confidence are unrelated to truth; verify important claims.
- 3'It remembers our past chats.' - By default it only sees the current context window unless a memory system supplies history.
- 4'Bigger is always better.' - Scale helps, but the right prompt, grounding, and a fit-for-purpose model often beat raw size.
- 5'It understands like a person.' - It has powerful statistical patterns of language, not human comprehension or intent.
Base model vs instruct model
| Aspect | Base (pretrained) | Instruct (post-trained) |
|---|---|---|
| Trained to | Predict next token | Follow instructions helpfully |
| Behavior | Continues text | Answers, chats, obeys format |
| Use | Foundation for tuning | What you actually chat with |
Sampling settings, intuitively
| Setting | Low | High |
|---|---|---|
| Temperature | Focused, repeatable | Varied, creative |
| Best for | Facts, code, extraction | Brainstorming, copy |
| Risk | Bland/rigid | Off-topic/hallucination |
Next-token prediction in action
Weak prompt
Thinking the model 'searches' for the answer to 'The capital of France is ___'.
Strong prompt
It tokenizes the prompt, runs it through the Transformer, and computes probabilities for the next token.
Output
P('Paris') is far higher than any other token, so it outputs 'Paris' - then would continue one token at a time.Why it hallucinates
Weak prompt
Asking for an obscure fact and trusting the confident reply verbatim.
Strong prompt
Recognize the model is predicting a plausible continuation; ground it with a source or ask it to cite.
Output
With no supporting context, a made-up but plausible citation can 'win' the prediction - so you verify or provide the source.
LLM vs traditional software
| Dimension | Traditional software | LLM |
|---|---|---|
| Behavior | Deterministic rules | Probabilistic prediction |
| Same input | Same output | May vary (sampling) |
| Fails by | Crashing/erroring | Confidently guessing |
| Fix by | Editing code | Better prompt/context/grounding |
Frequently asked questions
Does an LLM understand language?
It has learned rich statistical patterns of language from vast text, which lets it produce remarkably coherent output - but under the hood it is predicting the next token, not reasoning from a stored knowledge base.
Why do LLMs hallucinate?
Because they generate the most likely continuation, not verified facts. When the training data is thin or the prompt is ambiguous, a plausible-sounding but wrong token sequence can win - which is why grounding and verification matter.
What is a token, really?
A token is a small chunk of text (often a word piece) that the model converts into a number. Models read and bill by tokens, and everything must fit inside the context window.
What is the context window?
The maximum amount of text (in tokens) the model can consider at once. Anything outside it is invisible to the model unless summarized or retrieved back in.
Why does the same prompt give different answers?
Because of sampling. At higher temperature the model chooses among likely next tokens more randomly; lower temperature makes it more repeatable.