Most frontier models now ship with a version of the same knob. Claude exposes extended thinking with a token budget, and coding harnesses wrap it in levels with names like low, medium, high, and max; OpenAI models take a reasoning_effort parameter. Behind every one of these labels sits the same quantity: how much private scratch work the model may do before it starts the answer you see.
Engineers tend to treat the knob superstitiously, either pinned at max "to be safe" or left untouched. It deserves better, because it maps to something mechanical, its returns have a knowable shape, and misjudging it is either money and minutes wasted or a wrong answer delivered confidently.
What the knob turns
The first piece in this series covered the constraint that makes effort interesting: a transformer spends a fixed amount of compute per token. One forward pass through the stack, roughly two FLOPs per parameter, whether the token completes boilerplate or carries the crux of a proof. There is no loop inside the architecture where a model could quietly think harder about a hard token. Depth is fixed at inference; width is fixed; one pass is one pass.
The only way to give a model more computation is to give it more tokens, and that is the entire mechanism. Raising effort raises the budget of thinking tokens: text the model generates for itself, conditioned on like any other output, but held back from the visible reply. Each thinking token is one more forward pass whose result lands back in the context, available to every pass that follows. Intermediate conclusions become inputs. A model working through a race condition can name the interleaving, check it against the lock order, discard it, and try another, and each of those steps is real computation the answer-so-far gets to build on.
Nothing else changes. Same weights, same knowledge, same context. Effort buys serial depth, one extra step at a time.
The shape of the returns
Spending is linear: token cost and wall-clock latency grow in direct proportion to the budget. The return is anything but. Quality as a function of thinking budget is concave, it climbs steeply while the model still has genuine steps to work through and flattens once it runs out of them. Where the flattening happens depends on the task, and that dependence is the entire skill of setting the knob.
Three regimes show up in practice, and the figure sketches one task from each:
- Flat from the start. A mechanical rename has no multi-step structure. The model's first read of the problem contains the answer, and additional thinking mostly restates it. Extra budget here buys latency, and occasionally worse: models given room to deliberate on trivial questions sometimes talk themselves out of a correct first take.
- Steep, then saturating. A mid-size feature rewards enough thinking to plan the change, check the interfaces it touches, and catch one or two interactions. Past that, the plan stops improving before the tokens stop flowing.
- Steep for a long time. A concurrency bug, a cross-cutting design decision, a security review: tasks whose answer sits behind a chain of dependent deductions keep paying for depth, because each thinking pass extends the chain instead of rephrasing it.
One more property matters: effort cannot add what the weights and the context lack. If the model has never seen your internal framework, no thinking budget will derive it. When a failure looks like missing knowledge, the fix is context or a different model, and when it looks like skipped steps, the fix is effort. Learning to tell those two failure smells apart pays for itself quickly.
Picking a level
Two questions do most of the work: how long is the chain of dependent steps between the problem and the answer, and what does a wrong answer cost net of how cheaply you can catch it? Shallow, self-verifying tasks go low. Deep ones, and any task expensive to be wrong about, go high. Everything else sits in the middle, which is why the middle is the default.
| Level | Reach for it when | Examples |
|---|---|---|
| low | the task is mechanical, reversible, and its output is checked by tooling anyway | renames, formatting, boilerplate, config edits, summarizing a diff |
| medium | ordinary engineering work with a few interacting parts; the daily default | a scoped feature, a straightforward bug, test writing, code review of a small change |
| high | the answer sits behind several dependent deductions, or spans systems | debugging heisenbugs, API design, performance work, security review |
| max | one shot at a correctness-critical answer, and verification is expensive | migration plans, concurrency invariants, incident root-cause under pressure |
Calibrate against the transcript. When the visible reasoning mostly restates the prompt, the budget is too high for the task; when an answer arrives fast with an unexamined assumption in the middle of it, the budget was too low. Both smells are easier to read than any rule of thumb.
Effort is not the same axis as model choice
Raising effort and upgrading the model both cost more, so they get conflated, and they should not be. Effort scales serial compute: more dependent steps on the same machinery. A stronger model improves the machinery itself: better priors and better judgment at every step. A small model at max effort will diligently explore a chain of wrong turns; a strong model at low effort will make one excellent snap judgment and miss an interaction it never looked for.
A third option competes with both: sampling. For tasks whose answers are cheap to verify, several low-effort attempts plus a checker can beat one max-effort attempt on both cost and reliability, since independent samples fail differently while one long deliberation fails as a unit. That pattern, spending tokens across attempts rather than within one, is where orchestration starts.
Defaults that hold up
- Set the session default to medium and deviate per task, in either direction, on the two-question test above.
- Spend effort where verification is weak. If a test suite will catch failure in seconds, low effort plus the suite outperforms max effort without it.
- Treat a max-effort run whose thinking you never sampled as unreviewed work product.
- When a high-effort answer still misses, stop raising effort. The bottleneck has moved to context or model, and more budget will explore the same gaps more thoroughly.