2 of 5 · The machine

How a language model turns text into the next token

A large language model (LLM) reads tokens and predicts one token that can come next. It adds that token to the text and runs again. Everything else in the machine exists to make that one prediction useful.

1 · CutText becomes tokens
2 · Look upTokens become vectors
3 · MixContext changes each vector
4 · ChooseOne token joins the text

About 18 minutes to read. Every figure responds to its controls. The formulas sit in sections that you can open.

Technical scope

The figures show a modern decoder-only transformer, the family that GPT-style models use. The original 2017 transformer had an encoder and a decoder. Model widths, layer counts, token IDs, probabilities, and positions on this page are teaching examples. The text gives the real tensor shapes and the order of operations where they matter.

Section 1 · Lifecycle

A model is built once, then used many times

In one sentenceTraining sets the model's numbers. A request reads those frozen numbers to produce one token at a time.

The build path collects text, creates a tokenizer, and adjusts billions of learned numbers called weights. Training saves the finished weights in a checkpoint.

The run path cuts a prompt into tokens, reads the frozen checkpoint, chooses one next token, and repeats. A normal request does not change the checkpoint.

Try it · 2 min

Press Next to follow the build path, then the run path. Note where the tokenizer and the checkpoint appear in both.

Step1 / 12

Corpus selectedFilters keep the useful documents and drop the rest. The model learns only from this text.

Back deck: build, onceFront deck: run, once per tokenType slugs: tokens, each with its IDSteps 1 to 7 build the model once. Steps 8 to 12 show the first pass of one request. Station sizes and counts are illustrative, and the token IDs come from the site's teaching tokenizer.

Section 2 · Corpus

The model learns from a selected dataset

In one sentenceA model can learn only the patterns that its training data contains.

The build path starts with text: web crawls, books, code, and papers. None of it is ready to use. Data pipelines extract the usable text, remove boilerplate and near-duplicates, apply quality and safety filters, and choose a mix of languages and domains. Then they split the documents into training examples. These choices decide which patterns the model can see, and how often.

Engineers usually train the tokenizer (Section 3) on a representative sample of the corpus. Then they use that frozen tokenizer to encode the much larger training set. The FineWeb paper publishes concrete filtering and deduplication experiments. Reports on proprietary models often disclose only part of the data recipe.

Accuracy check

Documents do not flow straight into a transformer. They become packed arrays of token IDs, usually with a boundary token where one document ends and the next begins. Figure 2 shows storage, shuffling, packing, and distributed loading as one line.

Sheets enter the line and pass the filters.

Lit: the station at work nowSheets: one document eachRows: 16 token IDs, a divider after each documentSchematic, not to scale. The counts are illustrative. The filters reject 8 of 20 documents. The tokenizer trains on a sample of 2 kept documents and then freezes. The frozen tokenizer then encodes all 12 kept documents, including the 2 in the sample. The packer cuts the token stream into 4 sequences of 16 tokens and puts a divider token after each document. The model trains on these rows of token IDs, not on the documents. Corpus choices decide what the model can learn.

Section 3 · Tokenization

Text becomes a sequence of vocabulary addresses

In one sentenceThe tokenizer cuts text into pieces and gives each piece a number.

A neural network computes with numbers, so text must become numbers first. A tokenizer cuts a string into pieces from a fixed vocabulary and returns their integer token IDs. A common word can be one token. A rare name can split into several pieces or single bytes. Spaces, punctuation, and fragments of code can be tokens too. To try it yourself, open Tokens are the meter.

Show the tokenizer notation

Tokenizer outputt₁, …, t_T = Tokenize(s), t_i ∈ {0, …, |V| − 1}

s is the input string, T is its token count, and |V| is the vocabulary size. A token ID is an address. Its numeric value carries no distance or meaning.

Byte-pair encoding (BPE) and Unigram tokenization choose the vocabulary with different algorithms. SentencePiece describes both in a language-independent system that trains from raw sentences. Production tokenizers often add byte fallback, so any character can be encoded, and special control tokens.

Keep the artifacts separate

Tokenizer training chooses a fixed vocabulary. Pre-training later learns the embedding vectors and the transformer weights. After pre-training starts, a change to the token-to-ID map breaks the embedding rows that belong to those IDs.

Try it · 1 min

Compare Sentence with Rare word and Emoji. Find the example that needs the most pieces.

The1211
␣old8762
␣light9054
house24801
␣watched31208
␣the4021
␣fog22104

Lit slug: the newest tokenSlugs: one token each, with its IDType case: one numbered compartment per token IDThis is a toy vocabulary. The boundaries and IDs are illustrative. An ID is an address in the vocabulary. It does not measure meaning. Different strings give very different token counts.

Section 4 · Embeddings and position

Each token ID selects a learned vector

In one sentenceEach token ID picks one row of numbers that the model can compare and change.

A token ID is only a catalog number. The embedding table gives it content: one learned row per vocabulary entry. A lookup of token t_i returns that row, a vector of d_model numbers. You can picture a vector as a point in space. During training, tokens that appear in similar contexts move toward nearby points. The vector from the lookup is the model's starting working state, its hidden state, at position i.

Show the embedding lookup

Embedding lookupx⁽⁰⁾_i = E[t_i], E ∈ ℝ^{|V| × d_model}

The bracket is a row lookup, not a search by meaning. Pre-training adjusts every row that the training text uses, through gradient descent.

Lit card: the row for the chosen tokenDrawers: one row each, in token ID orderSliders: one value each, above or below the zero lineThe embedding table has one learned row for each vocabulary entry. The token ID is the row number. The lookup reads that row directly and does not search. The row becomes the token's starting hidden state at its position. Pre-training adjusts the rows. Tokens that occur in similar contexts get similar rows. The cards for “light” and “lamp” almost match, but the cards for “fog” and “house” differ. The sizes are illustrative: |V| = 50,257 rows and d_model = 4,096 values. The cabinet shows 8 rows, and each card shows 8 values. The values are illustrative too.

The network also needs word order: "dog bites man" and "man bites dog" contain the same tokens. Many current models use rotary position embeddings (RoPE). In each attention layer, RoPE rotates pairs of coordinates in every query and key by an angle that grows with the token's position. After the rotation, the attention score between two tokens depends on the distance between their positions, not on where they sit in the text. RoPE does not add a vector to the embedding.

Show the RoPE notation

RoPE, simplifiedq′_i = R(i)q_i, k′_j = R(j)k_j

R(i) rotates each pair of coordinates by the angle i·θ, where each pair has its own frequency θ. The dot product q′_i · k′_j then depends on the offset i − j. See RoFormer.

A real hidden state can have thousands of coordinates. Nearby vectors suggest similar use, but the values in Figure 4 are examples, not measurements from a trained model. After the first layer, a hidden state also carries context, so it is not a fixed word meaning.

Section 5 · Transformer block

Attention mixes positions; the MLP transforms each one

In one sentenceAttention pulls useful context from earlier tokens, then the MLP updates each token's working state.

A transformer is a stack of nearly identical blocks. Each block makes two moves. Attention lets every position read from earlier positions, so the state for "bank" in "the river bank" can take in "river". The MLP, a small feed-forward network, then reworks each position on its own. The rest of this section states both moves exactly.

Let X hold the hidden states of a sequence. One attention head makes three linear projections. Queries describe what each position looks for. Keys describe what each position offers for matching. Values carry the information that attention mixes.

Show the attention math

One causal attention headQ = XW_Q, K = XW_K, V = XW_V
A = softmax(QK^T / √d_head + M_causal)
Attention(X) = AV

Row i of A holds the weights that update token i, and they sum to 1. M_causal[i, j] is −∞ when j > i, so softmax gives later positions a weight of zero. The division by √d_head keeps the dot products in a stable range.

Several heads run in parallel. The block combines their results, and an output projection returns them to the model width. Grouped-query attention lets heads share keys and values to reduce memory traffic. Each query still obeys the causal mask.

A modern block normalizes before each sublayer and adds the result back to a residual path:

Show the block equations

One decoder block, schematicx ← x + Attention(RMSNorm(x))
x ← x + MLP(RMSNorm(x))

The residual path keeps the current state while each sublayer writes an update. A SwiGLU-style MLP expands each position, gates it, then projects it back to d_model. Models differ in their exact normalization, gating, and bias choices.

Attention Is All You Need introduced the encoder-decoder transformer, with normalization after each sublayer. Modern decoder-only models often use RMSNorm, RoPE, gated MLPs, and grouped-query attention. OpenELM documents one compact example.

Tensor shapes in one forward pass
Object	Typical shape	What it holds
Token IDs	`[T]`	One ID per position; T is the sequence length
Hidden states	`[T, d_model]`	One vector per position
Attention scores	`[heads, T, T]`	One causal matrix per query head
Output logits	`[T, \|V\|]`	One score per vocabulary entry, per position

Try it · 1 min

Start with Full block, then pick Attention and MLP. Residual shows what each part adds to the running state.

Lane

Attention weights
the
river
bank
flooded
Sum

Lit: the part that works in this modeBlocks: one token's state on its own laneThin plates: the updates that attention and the MLP addStop plates: they cover the jacks of later lanesEach lane carries one token's state past attention and the MLP. Each part adds one update plate to the state. A lane's cords reach only its own lane and earlier lanes. Cord thickness shows the weight. The weights for each lane sum to 1. One die drives all four presses, so the MLP applies the same weights to each lane on its own. A model stacks dozens of these blocks. The weights are illustrative. Real implementations store these values in tensors and fuse many steps on accelerators.

Section 6 · Pre-training

Every position supplies a training target

In one sentenceThe model guesses each next token, measures each error, and adjusts its weights.

Pre-training sets the numbers in the block. It is a guessing game, graded at every position. The model reads corpus text, and at each position it must give a high probability to the token that comes next. The causal mask stops it from reading later tokens, so training can grade all positions of a sequence at the same time. This setup is called teacher forcing.

Show the training loss

Next-token model and cross-entropy lossp_θ(t_1:T) = ∏_i=1^T p_θ(t_i | t_<i)
L(θ) = −(1/N) Σ_i log p_θ(t_i | t_<i)

If the model gives the actual next token a probability of 0.80, that position adds −log(0.80) ≈ 0.223 to the loss. A probability of 0.05 adds about 2.996. A confident mistake costs more.

Backpropagation computes the gradient of the batch loss: for each weight, how a small change raises or lowers the loss. An optimizer such as AdamW keeps running statistics of these gradients and uses them to make a small update. Large runs spread this work across many accelerators.

Show the optimizer step

Optimizer step, schematicθ_k+1 = AdamW(θ_k, ∇_θL, η)

θ holds the embeddings, the attention projections, the MLP weights, the normalization scales, and the output projection. η is the learning rate. This schematic leaves out AdamW's moment estimates and weight decay.

The GPT-3 paper is a concrete reference for decoder-only training. Chinchilla found that, for a fixed compute budget, models did better when parameter count and training tokens grew together. That result is an empirical recipe for the regime it studied, not a law for every model or dataset.

After pre-training

Instruction tuning and preference training adjust the checkpoint again with curated examples and human or AI preferences. They shape behavior. Generation still uses the same forward pass, one token at a time. See InstructGPT for an influential example.

Watch · 1 min

Press Step. Find the targets shifted by one token, the losses at every position, and the error that walks back to the weights.

The → old: p –−log p –
old → light: p –−log p –
light → house: p –−log p –
house → watched: p –−log p –
watched → the: p –−log p –
the → fog: p –−log p –

Gauge and column at each position: p and −log pSumming beam and loss dial: the batch loss, the mean of the columnsReturn wire: carries the error back to the weightsTrim screws: the weights. Each step turns them a little.The bench runs one optimizer step at toy scale. The targets come from the text itself. The target at each position is the next token. The model scores all six positions at once, under the causal mask. The loss at each position is −log p. The batch loss is the mean of the six losses. Backpropagation finds the gradient for every weight. AdamW then makes a small update, so the next loss is a little lower. The probabilities are illustrative. Production runs use many sequences and many layers.

Section 7 · Generation

The last hidden state becomes one score per token

In one sentenceThe model scores every possible next token, picks one, adds it to the text, and runs again.

At request time, the frozen tokenizer encodes the prompt. The frozen transformer runs the same forward pass as in training, with no update afterward. A final normalization and an output projection turn the last hidden state into one logit, an unnormalized score, for every vocabulary entry. Some models reuse the embedding table as this output matrix.

Show logits and temperature

Logits and temperature-scaled probabilitiesz = W_Uh_T + b
p_i = exp(z_i/τ) / Σ_j exp(z_j/τ)

W_U is the output (unembedding) matrix; many models have no bias b. A lower temperature τ moves probability toward the largest logits. A higher τ flattens the distribution. Greedy decoding takes the largest logit. Top-k and top-p sampling first cut the candidate list, then sample from what remains.

The loop appends the chosen token, and decoding repeats. In the first pass over the prompt, called prefill, the model computes keys and values for every prompt token. A KV cache keeps them. Each later step then computes only the new position and attends over the cached keys and values. Systems such as PagedAttention manage that cache efficiently.

What stays fixed

The checkpoint does not learn from a request. The KV cache is temporary state for one sequence, not memory. Randomness enters only through the decoding rule. With fixed token IDs and weights the forward pass is deterministic in principle; in practice, batching and floating-point order can change results slightly.

Try it · 1 min

Generate one token. Then choose Top-k and raise the temperature. Compare how the probabilities spread.

Temperature τ0.8

Context

Candidates for the next token

zpdraw

Ball: the one random drawPockets: as wide as each probabilityShutters: pockets the rule removesColumns: one logit per candidateCandidate values are illustrative. The model computes every logit with one matrix multiplication. The figure shows eight candidates. In the table, z is the logit. The p value is exp(z/τ) divided by the sum of exp(z/τ) for the eight candidates. The draw value is the chance after the rule. Greedy takes the largest p. Top-k and top-p remove candidates, then divide each remaining p by their sum. The KV cache is temporary request state. The checkpoint stays frozen.

Section 8 · Complete pass

Put the whole machine back together

In one sentenceThe build path makes reusable artifacts once; the run path reads them once per generated token.

The build path turns a corpus into a tokenizer and, through pre-training, into a checkpoint. The run path uses that frozen tokenizer and checkpoint to turn a prompt into one selected token. The token extends the context, and the next step reads it too.

Go back to Figure 1 and step through all 12 stages again. Each station now matches a section of this page.

References

Papers and visual references

Links appear next to the claims that they support. This list collects the main technical papers and the visual work that shaped this page.

Attention Is All You NeedVaswani et al., 2017. Scaled dot-product attention and the original encoder-decoder transformer.
SentencePieceKudo and Richardson, 2018. BPE and Unigram tokenization trained directly from raw sentences.
The FineWeb DatasetsPenedo et al., 2024. A documented web corpus with filtering and deduplication experiments.
RoFormerSu et al., 2021. Rotary position embeddings applied to attention queries and keys.
Language Models are Few-Shot LearnersBrown et al., 2020. Decoder-only pre-training at GPT-3 scale.
Training Compute-Optimal Large Language ModelsHoffmann et al., 2022. Scaling of parameters and training tokens under a fixed compute budget.
OpenELMMehta et al., 2024. A documented modern decoder with RMSNorm, RoPE, grouped-query attention, and SwiGLU.
InstructGPTOuyang et al., 2022. Instruction tuning and training from human preferences.
PagedAttentionKwon et al., 2023. Efficient serving through paged management of the KV cache.
The Illustrated TransformerJay Alammar. A step-by-step visual explanation of the original transformer.
Transformer ExplainerCho et al., 2024. Interactive moves between whole-system and single-operation views of a transformer.
LLM VisualizationBrendan Bycroft. A detailed 3D walk through the forward pass of a small GPT.