How an LLM turns text into the next token

Section 2 · Corpus

The model's evidence starts as a selected dataset

In one sentence: the model can learn only from patterns that its training data contains.

The build loop starts with text. Training text arrives as web crawls, books, code, and papers, and none of it is ready to use. Data pipelines extract the usable text, remove boilerplate and near-duplicates, apply quality and safety filters, choose language and domain mixtures, then split documents into training examples. Those decisions determine what patterns the model can observe and how often it observes them.

Engineers usually train the tokenizer, described next in Section 3, on a representative sample of the corpus. They then use that frozen tokenizer to encode the much larger training set. The FineWeb paper is useful because it publishes concrete filtering and deduplication experiments. Proprietary model reports may disclose only part of the data recipe.

Accuracy check. Documents do not flow directly into a transformer during training. They become packed arrays of token IDs, commonly with boundaries or special tokens marking document transitions. Figure 2 compresses storage, shuffling, packing, and distributed loading into one stream.

Section 3 · Tokenization

Text becomes a sequence of vocabulary addresses

In one sentence: the tokenizer cuts text into pieces and assigns each piece a number.

The corpus supplies raw strings. A neural network computes with numbers, so text has to become numbers first. A tokenizer segments a string into pieces found in a fixed vocabulary and returns the corresponding integer IDs. A common word may occupy one token. A rare name may split into several subwords or bytes. Spaces, punctuation, and code fragments can be tokens too. To feel this in your hands, train and operate a tokenizer in the playground.

Byte-pair encoding (BPE) and Unigram tokenization use different algorithms to choose the vocabulary. SentencePiece describes both in a language-independent system that can train from raw sentences. Production tokenizers often add byte fallback and special control tokens.

Keep the artifacts separate. Tokenizer construction chooses a discrete vocabulary. Pre-training later learns continuous embedding vectors and transformer weights. Once model training begins, changing the token-to-ID map would invalidate the embedding rows tied to those IDs.

Section 4 · Embeddings and position

Each token ID selects a learned vector

In one sentence: each token number selects a list of values the model can compare and change.

A token ID by itself is only a catalog number. The embedding table attaches content to it: one learned row per vocabulary entry. Looking up token t_i returns that row, a vector of d_model numbers, and you can picture a vector as a point in space. Tokens that appear in similar contexts tend to drift toward nearby points during training. The looked-up vector is the initial hidden state, the model's working representation, at position i.

The network also needs token order; "dog bites man" and "man bites dog" contain the same tokens. Many current decoders use rotary position embeddings, or RoPE. RoPE rotates query and key coordinates at each attention layer, so attention scores depend on relative offsets between positions. It is not an extra point appended to the token embedding.

Figure 4 projects a toy set of vectors into three dimensions. A real hidden width may contain thousands of coordinates. Nearby points can suggest similar usage, but this drawing is not a measurement from a trained model. A contextual hidden state is not a static word meaning.

Section 5 · Transformer block

Attention mixes positions; the MLP transforms each position

In one sentence: attention pulls useful context from earlier tokens, then the MLP updates each token's working state.

A transformer is a stack of near-identical blocks, and each block makes two moves. Attention lets every position read from earlier positions, so the state for "bank" in "the river bank" can absorb "river". The MLP, a small feed-forward network applied to each position on its own, then reworks whatever attention gathered. The rest of this section states both moves precisely.

Let X hold the hidden states for a sequence. One attention head makes three linear projections. Queries represent what each position looks for, keys represent what each position offers for matching, and values carry the information that attention will mix.

Multiple heads run in parallel, the block combines their results, and an output projection returns to the model width. Grouped-query attention shares key and value heads to reduce memory traffic, but each query still obeys the causal mask.

A modern pre-normalized block commonly follows this residual pattern:

The original Attention Is All You Need paper introduced the encoder-decoder transformer with post-normalization. Modern decoder-only implementations often use RMSNorm, RoPE, gated MLPs, and grouped-query attention; OpenELM documents one compact example.

Section 6 · Pre-training

Every position supplies a next-token training target

In one sentence: the model guesses the next token, measures each error, and adjusts its weights.

With the block defined, the build loop can set its numbers. Pre-training is a guessing game graded at every position. The model reads corpus text and, at each position, must assign high probability to the token that comes next. The causal mask keeps it from peeking at later tokens, yet training can grade all positions in a sequence in parallel. This setup is often called teacher forcing.

Backpropagation computes the gradient of the batch loss: for every participating parameter, a measure of how a small change would raise or lower the loss. An optimizer such as AdamW maintains running statistics and uses that signal to apply a small update. Data-parallel and tensor-parallel systems distribute the work across many accelerators.

The GPT-3 paper is a concrete decoder-only training reference. Chinchilla showed that, within its studied compute regime, better results came from balancing parameter count with more training tokens. That scaling result is an empirical recipe, not a law that fixes every model or dataset.

After base training. Instruction tuning and preference optimization adjust the checkpoint again using curated examples and preference signals. They shape behavior, but generation still uses the same autoregressive forward pass. See InstructGPT for an influential pipeline.

Section 7 · Generation

The last hidden state becomes one score per token

In one sentence: the model scores possible next tokens, chooses one, adds it to the message, and runs again.

Sections 4 and 5 assembled the block, and Section 6 set its numbers. At request time, the tokenizer encodes the prompt, and the frozen transformer runs the same forward pass used in training, but with no update afterward. A final normalization and linear output projection map the latest hidden state to one logit, an unnormalized score, for every vocabulary entry. Some models tie this output matrix to the input embedding table.

The loop appends the chosen token, and decoding repeats. During the first prompt pass, called prefill, the model computes keys and values for every prompt token. A KV cache keeps those tensors, so later decode steps compute the new position and attend over cached keys and values. Systems such as PagedAttention focus on managing that cache efficiently.

What remains fixed. The checkpoint does not learn from the request. The KV cache is temporary sequence state, not durable memory. Randomness enters through the decoding rule; the forward pass for fixed token IDs and weights is deterministic up to numerical implementation details.