Model Mechanics Explainer 1

Decoder-only language models

How an LLM turns text into the next token

A language model has one job: given some text, predict a small piece of what comes next. Everything an LLM produces, from a chat reply to working code, is that prediction repeated. This article follows the full lifecycle in order: the corpus and tokenizer that prepare the data, the transformer block that computes on it, the pre-training that sets the numbers, and the generation loop that turns a prompt into text one token at a time.

Reading time: about 18 minutes · Interactive figures can be paused, replayed, and scrubbed

Scope. The diagrams show a modern decoder-only transformer, the family used by GPT-style language models. The 2017 transformer had an encoder and a decoder. Model widths, layer counts, token IDs, probabilities, and 3D positions here are illustrative; tensor shapes and operation order are stated where they matter.

Section 1 · Lifecycle

A model is built once, then used many times

An LLM has two separate lives. In the build loop, a data pipeline assembles a large collection of text called the corpus, a tokenizer turns that text into token sequences, and pre-training adjusts the model's parameters, often called weights: the billions of learned numbers that determine its behavior. The finished parameters are saved as a checkpoint, one large file the model can be reloaded from.

In the use loop, a prompt passes through the frozen tokenizer and checkpoint. The model produces one token, appends it to the sequence, and runs again, once per small piece of generated text. The parameters do not change while an answer is generated.

The sections below follow the build order: corpus, tokenizer, vectors and the transformer block, then pre-training, then the generation loop a prompt travels through. Training comes after the block because the update rule is easier to picture once you have seen what the parameters compute. Jay Alammar's Illustrated Transformer uses a similar progressive reveal for the original architecture.

Figure 1 One machine builds artifacts; another runs them frozen Building the corpus
Collect and filter a corpus Selected documents supply the evidence for tokenizer construction and pre-training.
1 / 6

Section 2 · Corpus

The model's evidence starts as a selected dataset

The build loop starts with text. Training text arrives as web crawls, books, code, and papers, and none of it is ready to use. Data pipelines extract the usable text, remove boilerplate and near-duplicates, apply quality and safety filters, choose language and domain mixtures, then split documents into training examples. Those decisions determine what patterns the model can observe and how often it observes them.

Engineers usually train the tokenizer, described next in Section 3, on a representative sample of the corpus, then use that frozen tokenizer to encode the much larger training set. The FineWeb paper is useful because it publishes concrete filtering and deduplication experiments. Proprietary model reports may disclose only part of the data recipe.

Accuracy check. Documents do not flow directly into a transformer during training. They become packed arrays of token IDs, commonly with boundaries or special tokens marking document transitions. Figure 2 compresses storage, shuffling, packing, and distributed loading into one stream.

Figure 2 Documents are filtered, then packed into token batches Collecting documents
sources
Schematic pipeline. Source proportions and rejection rates are illustrative. Corpus composition is a modeling decision, not background plumbing.

Section 3 · Tokenization

Text becomes a sequence of vocabulary addresses

The corpus supplies raw strings. A neural network computes with numbers, so text has to become numbers first. A tokenizer segments a string into pieces found in a fixed vocabulary and returns the corresponding integer IDs. A common word may occupy one token. A rare name may split into several subwords or bytes. Spaces, punctuation, and code fragments can be tokens too.

Tokenizer output
t₁, …, tT = Tokenize(s),   ti ∈ {0, …, |V| − 1}

s is the input string, T is its token count, and |V| is the vocabulary size. A token ID is an address. Its numeric value carries no distance or meaning.

Byte pair encoding and Unigram tokenization use different algorithms to choose the inventory. SentencePiece describes both in a language-independent system that can train from raw sentences. Production tokenizers often add byte fallback and special control tokens.

Keep the artifacts separate. Tokenizer construction chooses a discrete vocabulary. Pre-training later learns continuous embedding vectors and transformer weights. Once model training begins, changing the token-to-ID map would invalidate the embedding rows tied to those IDs.

Figure 3 A typesetter turns strings into addressed tiles Scanning a sentence
Toy vocabulary. Boundaries and IDs are illustrative. A wall slot encodes only an ID; neighboring slots imply no similarity. Different strings can produce widely different sequence lengths.

Section 4 · Embeddings and position

Each token ID selects a learned vector

A token ID by itself is only a catalog number. The embedding table attaches content to it: one learned row per vocabulary entry. Looking up token ti returns that row, a vector of dmodel numbers, and a vector can be pictured as a point in space. Tokens that appear in similar contexts tend to drift toward nearby points during training. The looked-up vector is the initial hidden state, the model's working representation, at position i.

Embedding lookup
x(0)i = E[ti],   E ∈ ℝ|V| × dmodel

The bracket is a row lookup, not a semantic database query. Pre-training adjusts every used row through gradient descent.

The network also needs token order; "dog bites man" and "man bites dog" contain the same tokens. Many current decoders use rotary position embeddings, or RoPE. RoPE rotates query and key coordinates at each attention layer, so attention scores depend on relative offsets between positions. It is not an extra point appended to the token embedding.

RoPE, simplified
q′i = R(i)qi,   k′j = R(j)kj

R(i) rotates pairs of coordinates by position-dependent angles. The dot product q′i · k′j then carries information about i − j. See RoFormer.

Figure 4 projects a toy set of vectors into three dimensions. A real hidden width may contain thousands of coordinates. Nearby points can suggest similar usage, but this drawing is not a measurement from a trained model, and a contextual hidden state is not a static word meaning.

Figure 4 Rows become vectors; position rotates pairs of coordinates Drag to rotate
Projection, not literal storage. The lattice is a three-dimensional shadow of a much higher-dimensional space, and the rotary view shows attention coordinates, not the embedding itself. The output layer later scores every vocabulary entry with matrix operations; it does not search this space for nearest points.

Section 5 · Transformer block

Attention mixes positions; the MLP transforms each position

A transformer is a stack of near-identical blocks, and each block makes two moves. Attention lets every position read from earlier positions, so the state for "bank" in "the river bank" can absorb "river". The MLP, a small feed-forward network applied to each position on its own, then reworks whatever attention gathered. The rest of this section states both moves precisely.

Let X hold the hidden states for a sequence. One attention head makes three linear projections. Queries represent what each position is looking for, keys represent what each position offers for matching, and values carry the information that will be mixed.

One causal attention head
Q = XWQ,   K = XWK,   V = XWV
A = softmax(QKT / √dhead + Mcausal)
Attention(X) = AV

Row i of A contains the weights used to update token i. Mcausal[i,j] is −∞ when j > i, so softmax assigns future positions zero probability. The division by √dhead keeps dot products numerically controlled.

Multiple heads run in parallel, their results are combined, and an output projection returns to the model width. Grouped-query attention shares key and value heads to reduce memory traffic, but each query still obeys the causal mask.

A modern pre-normalized block commonly follows this residual pattern:

One decoder block, schematic
x ← x + Attention(RMSNorm(x))
x ← x + MLP(RMSNorm(x))

The residual path preserves the current state while each sublayer writes an update. A SwiGLU-style MLP expands, gates, then projects each position back to dmodel. Exact normalization, gating, and bias choices vary by model.

The original Attention Is All You Need paper introduced the encoder-decoder transformer with post-normalization. Modern decoder-only implementations often use RMSNorm, RoPE, gated MLPs, and grouped-query attention; OpenELM documents one compact example.

A practical shape ledger
ObjectTypical shapeWhat varies
Token IDs[T]Sequence length T
Hidden states[T, d_model]One vector per position
Attention scores[heads, T, T]One causal matrix per query head
Output logits[T, |V|]One vocabulary score vector per position
Figure 5 One decoder block, exploded for inspection Following the full block
Exploded tensor view. The planes make data dependencies visible. Implementations store these values in tensors and fuse many operations into accelerator kernels.

Section 6 · Pre-training

Every position supplies a next-token training target

With the block defined, the build loop can set its numbers. Pre-training is a guessing game graded at every position. The model reads corpus text and, at each position, must assign high probability to the token that comes next. The causal mask keeps it from peeking at later tokens, yet all positions in a training sequence can be graded in parallel; this setup is often called teacher forcing.

Autoregressive model and cross-entropy loss
pθ(t₁:T) = ∏i=1T pθ(ti | t<i)
L(θ) = −(1/N) Σi log pθ(ti | t<i)

If the model assigns the observed token probability 0.80, that position contributes −log(0.80) ≈ 0.223. Probability 0.05 contributes about 2.996. Confident mistakes cost more.

Backpropagation computes the gradient of the batch loss: for every participating parameter, a measure of how a small change would raise or lower the loss. An optimizer such as AdamW maintains running statistics and uses that signal to apply a small update. Data-parallel and tensor-parallel systems distribute the work across many accelerators.

Optimizer step, schematic
θk+1 = AdamW(θk, ∇θL, η)

θ contains embeddings, attention projections, MLP weights, normalization scales, and the output projection. η is the learning rate. AdamW's moment estimates and weight decay are suppressed here.

The GPT-3 paper is a concrete decoder-only training reference. Chinchilla showed that, within its studied compute regime, better results came from balancing parameter count with more training tokens. That scaling result is an empirical recipe, not a law that fixes every model or dataset.

After base training. Instruction tuning and preference optimization adjust the checkpoint again using curated examples and preference signals. They shape behavior, but generation still uses the same autoregressive forward pass. See InstructGPT for an influential pipeline.

Figure 6 Score every position in parallel, then walk the error back Packing a token batch
forward pass
One optimizer step at toy scale. The targets come from the text itself: the target at each position is the next token. All positions are scored in parallel under the causal mask; nothing here generates one token at a time. Production runs use many sequences and layers, accumulate gradients, synchronize workers, and save checkpoints throughout training.

Section 7 · Generation

The last hidden state becomes one score per token

Sections 4 and 5 assembled the block, and Section 6 set its numbers. At request time, the prompt is tokenized and passed through the frozen transformer: the same forward pass used in training, but with no update afterward. A final normalization and linear output projection map the latest hidden state to one logit, an unnormalized score, for every vocabulary entry. Some models tie this output matrix to the input embedding table.

Logits and temperature-scaled probabilities
z = WUhT + b
pi = exp(zi/τ) / Σj exp(zj/τ)

WU is the output, or unembedding, matrix; some architectures omit the bias b. Lower positive temperature τ concentrates mass on larger logits. Higher τ flattens the distribution. Greedy decoding chooses the largest logit; top-k and top-p sampling first restrict the candidate set.

The chosen token is appended and decoding repeats. During the first prompt pass, called prefill, the model computes keys and values for every prompt token. A KV cache keeps those tensors so later decode steps compute the new position while attending over cached keys and values. Systems such as PagedAttention focus on managing that cache efficiently.

What remains fixed. The checkpoint does not learn from the request. The KV cache is temporary sequence state, not durable memory. Randomness enters through the decoding rule; the forward pass for fixed token IDs and weights is deterministic up to numerical implementation details.

Figure 7 Score the whole vocabulary, choose one tile, append it Scoring the vocabulary
0.8
The skyline is visual shorthand. The model computes all logits with one matrix multiplication; the magnifier expands a few entries for inspection. The KV cache is temporary request state; the checkpoint stays frozen. Candidate values are illustrative.

Section 8 · Complete pass

Put the data, weights, and token loop on one timeline

The upper route builds the artifacts: a corpus supports tokenizer construction and pre-training, then training saves a checkpoint. The lower route uses the fixed tokenizer and checkpoint to map a prompt to one selected token. That token extends the context for the next decode step.

Figure 8 Corpus to checkpoint, prompt to next token Training text enters the pipeline
Training begins with selected text The corpus supplies both the tokenizer sample and the encoded sequences used for next-token training.
1 / 12
cobalt: data coral: learning gold: output

References

Papers and visual references

Links appear beside the claims they support. This list collects the primary technical references and the visual work that informed the article's pacing.