Training gives a language model scores for possible next tokens. Generation is the policy that turns those scores into one token, appends it to the context, and repeats. That policy is often treated as a handful of magic numbers. It is better understood as a sequence of explicit transformations with measurable consequences.
This tutorial uses a deterministic trigram model so the sampling layer stays visible. The model is not the point. Temperature, top-k, top-p, and stopping rules operate on logits from any model. By separating the source of the logits from the decoding policy, we can test the policy without a GPU or a hosted account.
Learning objectives
You will implement stable softmax, temperature scaling, top-k, nucleus (top-p), and min-p filtering. You will measure entropy and survivor counts, generate repeated samples under fixed seeds, and design a bounded policy for a customer-support assistant.
The example is in content/foundations/temperature-top-k-top-p.examples. It uses Python 3.11+ and NumPy, runs in under a second on the recorded machine, and has no network dependency.
Prerequisites
You should understand next-token probabilities from the preceding article and basic logarithms. No neural-network framework is required.
Temperature changes relative probability
If logits are z, temperature T produces:
pᵢ = softmax(zᵢ / T)At T = 1, the original distribution is preserved. Lower temperatures increase the separation between high and low logits. Higher temperatures flatten it. The order remains unchanged as long as T is positive. Temperature therefore changes uncertainty, not the model’s ranking of alternatives.
Temperature is not a creativity dial with a universal “good” setting. A low value can make a support reply stable but brittle. A high value can make a brainstorming response varied but introduce unsupported claims. The right setting depends on the task, the acceptable error, and the constraints applied after sampling.
Toy problem / implementation
The local model is a smoothed trigram distribution trained on 60 ticket-like sentences. It exposes logits, and the decoder applies transformations in a fixed order. Temperature is applied before filtering. Top-k keeps the k highest-scoring tokens. Top-p sorts tokens, accumulates probability, and keeps the smallest prefix whose cumulative probability reaches p. Min-p keeps tokens above a fraction of the highest probability.
Filtering must renormalize the survivors before sampling. Otherwise the remaining values are not a probability distribution. The implementation also handles temperature=0 as greedy selection and uses a seeded NumPy generator for reproducibility.
Run it yourself
From the repository root:
cd content/foundations/temperature-top-k-top-p.examples
./verify.shThe verifier runs 46 tests and the demo. It checks probability normalization, temperature monotonicity, truncation behavior, stop sequences, end-of-text handling, deterministic seeds, and diversity measurements.
After thanks for, the model’s most likely continuation is reaching. At temperature 0.5 it receives 0.7184 probability; at 1.0 it receives 0.3927; at 1.5 it receives 0.1501. The entropy rises from 1.190 to 3.771 to 6.853 bits. These are the bundled model’s values, not general claims about all language models.
Top-k and top-p are different controls
Top-k fixes the number of candidates. k=5 always leaves five survivors, whether the fifth token is nearly impossible or genuinely plausible. That predictability can be useful for a tightly constrained task, but it ignores the shape of the distribution.
Top-p adapts to the distribution. On a peaked context, a small number of tokens may reach 0.9 cumulative probability. On a flat context, many more are needed. The example’s top-p count therefore changes with temperature and prompt while top-k remains constant.
Min-p is another adaptive rule: it keeps tokens whose probability is at least a fraction of the top token’s probability. It can remove a long tail while preserving a variable number of alternatives. Each rule encodes a different risk preference; none guarantees factuality.
Stopping is part of correctness
Generation should stop on an end-of-text token, a configured stop sequence, or a maximum token count. A maximum is a safety boundary even when the model fails to produce an end token. Multi-token stop sequences must be detected across token boundaries, and the stop text should usually be removed from the returned content.
Test stopping independently from sampling. A seeded test that happens to stop early does not prove the max-token branch works. The bundled tests force each stop condition and verify the returned reason.
Real-world application: controlled support generation
For a customer-support assistant, start with a low-variance policy for policy-sensitive replies. Ground the answer in retrieved policy text, constrain the output schema, set a maximum length, and stop on the format boundary. If the request is an exact extraction or classification, do not sample when a deterministic parser or classifier is sufficient.
Use a higher temperature only where multiple valid phrasings are acceptable, and evaluate it on a fixed set. Track refusal rate, unsupported-claim rate, formatting failures, duplicate phrases, and human edits. Store the model version, decoding parameters, seed policy, and retrieved context identifiers with each evaluation trace.
Sampling cannot compensate for poor retrieval or an incorrect model distribution. If the answer is unsupported at temperature 0, increasing temperature does not create evidence. It only changes which unsupported continuation is selected.
A practical tuning policy
Treat decoding as a policy per task, not a global preference. For exact classification, extraction, tool arguments, and short policy statements, start with greedy decoding or a very low temperature and validate the result against a schema. For grounded explanatory prose, start near temperature 0.2–0.7, use a modest top-p, and measure unsupported claims. For brainstorming, allow more entropy but keep a length limit and make it clear that the output is a set of candidates rather than an approved answer.
Do not tune from one prompt. Build a small matrix with easy, ambiguous, long, adversarial, and empty-context cases. Run the same prompts across candidate settings. Record not only the text but the token count, stop reason, latency, model version, retrieved evidence, and validation result. A setting that looks fluent on a demo prompt can fail on a long conversation because the context changes the distribution.
If you need reproducible tests, use a local model or a provider mode that documents seed behavior and compare semantic assertions rather than exact strings when backend nondeterminism is expected. Exact snapshots are useful for deterministic local components, but they can become brittle for a hosted model whose serving stack changes. Keep both kinds of tests separate.
The support workflow should also define what happens when generation fails. A malformed structured response can be retried once with a validation error represented as machine-readable feedback, but retries need a cap. If the model repeatedly fails, fall back to a template, return a human-review task, or ask the user for clarification. A retry loop that silently increases tokens is an operational failure, not resilience.
Temperature and truncation interact with calibration. A probability of 0.8 after temperature scaling is not automatically an 80 percent chance that the entire response is correct. It is a probability assigned to the next-token distribution under a transformed score vector. Do not expose it as user-facing confidence without a separate calibration study on the actual task.
Failure modes and debugging
- Sampling logits as probabilities. Apply softmax first, or use a numerically stable logit sampler.
- Filtering without renormalization. The survivors must sum to one before drawing.
- Top-p off-by-one errors. Include the token that crosses the cumulative threshold, then stop.
- Temperature applied after sampling. Temperature transforms the distribution before the draw.
- Seed assumptions. A fixed seed reproduces one local generator sequence, not every provider’s backend.
- No maximum length. A missing end token can create an unbounded loop and cost surprise.
- Comparing samples instead of distributions. Measure entropy and survivor counts as well as text.
Limitations / when not to use
This experiment uses a tiny n-gram model and short tickets. Its diversity numbers are educational measurements, not a production benchmark. Provider APIs may add server-side sampling, hidden reasoning, batching, or nondeterminism that changes exact reproducibility.
Do not use temperature to solve a factuality problem, top-p to solve authorization, or a stop string as a substitute for output validation. Use schemas, parsers, retrieval checks, permission boundaries, and human review where the consequence requires them.
The implementation in slow motion
The decoder first obtains one logit per vocabulary item. It divides the vector by temperature, subtracts the maximum for numerical stability, exponentiates, and normalizes. Filtering then marks disallowed entries with negative infinity or removes them from a candidate list. The final probabilities are normalized again, and the seeded generator draws one index. The chosen token is appended to the context, and the loop repeats.
That order is worth testing because small reorderings change the behavior. Applying top-k before temperature usually leaves the same rank set for positive temperatures, but the probabilities inside the set still depend on temperature. Applying top-p after an incorrect normalization can keep the wrong number of tokens. Sampling before filtering is not filtering at all. A short test with a hand-written three-token distribution can establish each invariant before a model is involved.
Entropy is a convenient summary of the distribution, defined as the negative sum of p log2(p). It does not tell us whether the top token is correct, but it does tell us how concentrated the distribution is. The demo reports mean and standard deviation across several prompts because one context can be sharply peaked while another is uncertain. Reporting only an average hides that variation.
Diversity metrics need the same caution. Distinct-1 and distinct-2 describe how many unique unigrams and bigrams appear in sampled text. A high score can mean useful variety or incoherence; a low score can mean repetition or a legitimately constrained answer. Pair the metric with task validity and human edits. A support assistant should not be rewarded for changing correct policy wording merely to increase diversity.
The safest default is therefore not “turn the temperature down.” It is to make the generation contract explicit: what outputs are allowed, which evidence must be present, how long the response may be, when to stop, how failures are retried, and when a person takes over. Decoding parameters are one layer of that contract.
That contract should be versioned and evaluated with the application continuously, before every production release cycle, explicitly, always.
Exercises
- Compare top-k and top-p survivor counts on every prompt in the demo.
- Add a no-repeat bigram constraint and measure its effect on diversity and validity.
- Build a deterministic JSON extraction policy and reject malformed output.
- Run 100 seeds and report confidence intervals for a repetition metric.
Next in this path
The next article is Fine-Tuning Language Models: Full Training, LoRA, and the Practical Middle Ground. It asks whether changing parameters is necessary at all, then compares full updates with smaller adapters in practice, carefully.
Sources
The research dossier is temperature-top-k-top-p. Nucleus sampling follows Holtzman et al., The Curious Case of Neural Text Degeneration, and top-k follows Fan et al., Hierarchical Neural Story Generation. Typical sampling is discussed by Meister et al., Locally Typical Sampling, while min-p follows Nguyen et al., Turning Up the Heat. Implementation defaults are cross-checked against the Hugging Face generation configuration documentation and the Anthropic Messages API documentation. Reproducibility uses NumPy’s Generator.
Research and code were last verified on 2026-08-05. All numerical claims belong to the deterministic bundled experiment.