The phrase “language model” can sound like a claim about language understanding. At its most basic, it describes a probability distribution over the next token given the tokens that came before it. Training supplies many such prefix-and-next-token examples, adjusts parameters to increase the probability of the observed next token, and repeats.
That simple objective is powerful enough to produce fluent behavior when the model, data, compute, and optimization are large enough. It is also easy to misunderstand. A model can lower next-token loss by learning spelling, formatting, and common sequences without acquiring a reliable world model. This tutorial keeps the objective visible with a deliberately small NumPy implementation.
Learning objectives
You will shift a sequence into inputs and labels, calculate cross-entropy and perplexity, train a tiny causal language model on 60 support tickets, compare it with a uniform baseline, and run a numerical gradient check. The example is in content/foundations/next-token-prediction.examples; it completes on a CPU without downloads.
The experiment is designed to make three boundaries clear: the labels come from the text itself, held-out loss is different from training loss, and pretraining is not the same thing as instruction-following or production adaptation.
Prerequisites
You need Python 3.11+, NumPy, basic probability, and the embedding and attention articles that precede this one. You do not need a GPU or a deep-learning framework. The code is intentionally more explicit than an optimized library implementation.
The shifted-label idea
Suppose a character sequence is access. The model sees a, then ac, then acc, and so on. In a training window, every input position has a label one position to its right. The label is not separately annotated by a human.
For a token sequence x₀, x₁, …, xₙ, the objective is the average negative log probability assigned to xᵢ₊₁ after seeing the prefix through xᵢ. A low loss means the model assigns high probability to the observed continuation. Cross-entropy is measured in nats when natural logarithms are used.
Perplexity is exp(loss). It can be read as an effective branching factor: a perplexity of 10 means the model is, very roughly, as uncertain as choosing uniformly among 10 options. It is useful for comparing the same tokenization and evaluation set, but it is not a universal quality score across different vocabularies or tasks.
Toy problem / implementation
The model is a small causal transformer-like network implemented in NumPy. It has token and positional embeddings, one self-attention block, a feed-forward block, layer normalization, and an output projection. The corpus is split by ticket so validation text does not simply duplicate the training window. Training uses seeded updates and records train and held-out loss.
The free baseline is important. If the vocabulary has V symbols, a uniform model has cross-entropy log(V) and perplexity V. Any model that performs at or above that baseline has not learned useful predictive structure under this measurement.
The example also checks its backward pass numerically. For a selected parameter, it nudges the value up and down, measures the loss difference, and compares the central finite difference with the analytic gradient. This does not prove every training run is correct, but it catches many sign, transpose, and normalization mistakes.
Run it yourself
From the repository root:
cd content/foundations/next-token-prediction.examples
./verify.shThe verifier runs 37 tests and the deterministic demo. The recorded environment is Python 3.13.13 and NumPy 2.4.5. The default run trains for 800 steps and takes about seven seconds on the recorded CPU.
The demo reports a 28-character vocabulary. Uniform guessing has perplexity 28.0. The best held-out perplexity is 8.12 at step 280, while the final step reaches 13.66 even though final training perplexity falls to 2.44. The divergence is the small corpus making overfitting visible, not a promise about how large models behave.
Inspecting the loss
Training loss falls nearly monotonically. Held-out loss improves early, bottoms out, and then rises. If we selected the final checkpoint because its training loss was lowest, we would ship the more memorized and less generalizing model. Save the best validation checkpoint or use a stopping rule tied to the intended evaluation set.
The model’s generated samples are revealing rather than impressive. After training, it produces fragments that have ticket-like punctuation and vocabulary but often contain malformed words. The model has learned surface statistics from 2,611 training characters; it has not learned a reliable support workflow or the truth of any ticket.
What the loss does and does not reward
Cross-entropy gives a large penalty to a confident wrong prediction. If the correct next token receives probability 0.9, its contribution is small; if it receives probability 0.001, the contribution is large. The objective therefore encourages probability mass on the observed continuation, not merely a correct top-one choice. A model that assigns useful probability to several plausible continuations can have better loss than a model that is occasionally certain and often wrong.
The training target is still only one observed continuation. Natural language often has many valid next tokens. A ticket may continue with the, our, or a punctuation mark depending on the writer. Teacher forcing gives the model the real previous tokens during training; generation feeds the model its own sampled tokens. Errors can therefore compound at inference time even when one-step validation looks reasonable.
Backpropagation turns the next-token error into parameter updates through the output projection and the shared representation. The gradient check in the example compares this chain of derivatives with a direct loss measurement. When the two agree, it gives us confidence that the implementation is optimizing the stated objective rather than an accidental variant. Numerical checks are especially valuable in educational code because a transpose can leave shapes valid while changing the calculation.
That habit generalizes to every custom training experiment.
Check a loss value independently, inspect one batch by hand, and keep a tiny deterministic fixture that fails loudly when a refactor changes the objective. Reproducibility is not just a seed; it is a complete chain from bytes and labels to gradients, checkpoints, and evaluation outputs.
This gap is why the sampling article follows this one. The model supplies logits, but temperature and truncation alter the distribution used for generation. A decoding setting cannot repair a model that has learned the wrong distribution, and a strong model can still produce poor output when decoding is unconstrained for a task that needs exact fields.
The toy run also includes a repetitive corpus where the model can nearly memorize the next character. Its loss approaches zero but does not become exactly zero because the first positions in a window have limited context. This is a useful reminder that a low loss can describe memorization of a narrow sequence. Always name the corpus and split when reporting a number.
From pretraining to post-training
Large language models apply the same next-token objective to far larger and more varied corpora. Scale changes what can be learned, but it does not remove data quality, licensing, memorization, evaluation, or safety concerns. Later stages can adapt a base model with supervised examples, preference data, or task-specific objectives. Instruction tuning changes the behavior expected from a user-facing assistant; it does not make the model omniscient.
When adapting a model to a domain, first decide whether the problem is knowledge, behavior, or retrieval. New changing facts often belong in retrieval rather than weights. A stable output format may be improved with supervised examples or structured decoding. A small classifier may be cheaper and more reliable than adapting a general language model.
Real-world application: a domain-adapted language system
A production team should define a held-out evaluation set before changing the model. Include representative user inputs, rare but important cases, refusal cases, and examples that test whether private or unsupported information appears. Record the tokenizer, model version, prompt or template, decoding settings, and evaluation code.
Use the base model’s loss as one diagnostic, not the release gate. A lower perplexity can coexist with worse instruction following, more repetition, weaker citation behavior, or unsafe completions. Evaluate the application task directly and compare multiple seeds or checkpoints when results are unstable.
The data pipeline deserves the same attention as the optimizer. Remove or quarantine secrets, personal data, duplicate documents, and evaluation examples that should remain held out. Track provenance and licensing decisions. Deduplicate before splitting when near-identical tickets or mirrored web pages could appear in both sets. A clean-looking validation number is not credible if the split leaks the answer.
For a domain adaptation project, keep a baseline that does not use a language model: keyword routing, a linear classifier, or retrieval with a template. Compare latency, memory, failure handling, and human correction cost as well as language quality. A model with a slightly lower loss is not automatically the better system if it is harder to audit or causes more costly mistakes.
Failure modes and debugging
- Off-by-one labels. Print a short sequence and assert that every label equals the next input symbol.
- Leaking the future. A causal mask must block positions after the current prediction.
- Training and validation overlap. Split by document or ticket before making windows.
- Only reporting training loss. Track held-out loss and task behavior from the beginning.
- Unstable softmax. Subtract the maximum logit before exponentiating.
- Confusing perplexity across tokenizers. Compare like with like.
- Treating a fluent sample as evidence. Samples are qualitative probes; use a fixed evaluation set for claims.
Limitations / when not to use
This model is too small and the corpus too narrow to say anything about general language modeling. It is a teaching instrument. Its validation set is small, its generated text is not a product, and its gradients are written for clarity rather than speed.
Do not fine-tune a language model when a deterministic template, retrieval system, or ordinary classifier solves the problem more cheaply and audibly. Do not put changing knowledge into weights merely because the model can memorize it. Do not treat a lower loss as permission to skip security, privacy, or human review.
Exercises
- Train for 1,600 steps and compare the best and final held-out checkpoints.
- Replace characters with the BPE tokenizer from the previous article and compare perplexity only after defining the new baseline.
- Add a duplicate-ticket split to test memorization.
- Ablate positional embeddings and inspect the loss.
- Compare greedy and sampled decoding in the next article.
Next in this path
The next article is How Language Models Generate Text: Temperature, Top-k, and Top-p. Training produces a distribution; decoding decides how to turn that distribution into a continuation.
Sources
The research dossier is next-token-prediction. The neural language-model objective follows Bengio et al., A Neural Probabilistic Language Model. GPT-2 and GPT-3 provide larger causal-language-model examples through Language Models are Unsupervised Multitask Learners and Language Models are Few-Shot Learners. Scaling and compute allocation are discussed by Kaplan et al. and Hoffmann et al.. Instruction adaptation follows InstructGPT. Degeneration concerns follow The Curious Case of Neural Text Degeneration. The numerical implementation uses NumPy’s Generator behavior and standard cross-entropy definitions.
Research and code were last verified on 2026-08-05. All quoted metrics come from the bundled deterministic run and should be regenerated if the corpus or default step count changes.