An embedding is a vector representation: a fixed-length list of numbers that a model can manipulate with linear algebra. The useful phrase is not “each number means something.” Usually it does not. The useful phrase is “relationships in the space can carry information.” Two vectors may be close because their inputs appear in similar contexts, because a predictive model learned similar behavior for them, or because a sentence encoder was trained to place related sentences near one another.
This tutorial builds the smallest useful version of that story. We count word contexts in a tiny support-ticket corpus, convert counts to positive pointwise mutual information, factor the matrix with singular value decomposition, and inspect the resulting two-dimensional vectors. Then we connect that toy to predictive word embeddings, sentence embeddings, semantic retrieval, and the operational risks of changing a model version.
Learning objectives
You will learn why context statistics produce geometry, what PPMI changes, why SVD can reduce a sparse matrix, and why a two-dimensional plot is an inspection tool rather than a proof of meaning. You will also run a local example in content/foundations/what-embeddings-encode.examples and compare same-intent and different-intent vectors.
The article makes a deliberately narrow promise. It explains where useful similarity can come from. It does not claim that an embedding is an objective semantic representation, that cosine similarity guarantees relevance, or that one vendor’s vector space can be mixed with another’s.
Prerequisites
You need basic Python, arrays, and the cosine-similarity idea from the preceding tutorial. NumPy is the only dependency. The example uses six tiny sentences and is designed to finish immediately on a CPU.
The distributional starting point
The distributional hypothesis is commonly summarized as words that occur in similar contexts tend to have similar meanings. A count-based method turns that intuition into a matrix. Rows are target words, columns are context words, and a cell records how strongly the pair is associated.
Raw counts are not enough. Common words appear near everything and can dominate the matrix. Pointwise mutual information compares the observed pair probability with the probability expected if the two words were independent:
PMI(w, c) = log2( P(w, c) / (P(w) P(c)) )Positive PMI clips negative values to zero. A negative association can be informative, but sparse text matrices often benefit from focusing on unexpectedly present pairs. This choice loses information, so it is a modeling decision rather than a law of language.
Toy problem / implementation
The example contains ticket-like phrases about refunds, invoices, login, accounts, exports, and reports. A symmetric window counts neighbors on both sides. After PPMI weighting, SVD decomposes the matrix into orthogonal directions. Keeping the first two dimensions gives a compact vector for plotting:
u, singular, _ = np.linalg.svd(matrix, full_matrices=False)
vectors = u[:, :dimensions] * np.sqrt(singular[:dimensions])SVD does not discover named dimensions such as “billingness.” It finds directions that explain variance in the matrix. Rotating the resulting coordinates can leave all pairwise relationships unchanged. The coordinates are therefore not individually sacred; distances and neighborhoods are the more useful objects.
The bundled demo prints a 12-by-12 PPMI matrix and two coordinates for every vocabulary item. It also computes cosine similarities. With a context window of two, refund and duplicate land in the same direction in this toy matrix, while refund and login point in different directions. That result is an illustration of the count construction, not a linguistic benchmark.
Run it yourself
From the repository root:
cd content/foundations/what-embeddings-encode.examples
./verify.shThe verifier runs three tests and the demo. Tests check symmetric window counts, PPMI clipping, SVD shape, and cosine self-similarity. The code uses Python 3.13.13 and NumPy 2.4.5 in the recorded run.
A useful experiment is to change the window from one token to two. The vector geometry changes because the definition of context changes. That is the point: an embedding encodes the evidence and choices of its training process. It does not contain a context-free dictionary definition of a word.
From counts to predictive embeddings
Word2vec-style skip-gram learns vectors by rewarding observed word-context pairs and penalizing sampled negative pairs. The training process looks different from PPMI plus SVD, but the relationship is not accidental. Levy and Goldberg showed that skip-gram with negative sampling can be understood as implicitly factorizing a shifted PMI matrix under assumptions about the objective.
That connection is useful and limited. It explains why two routes can produce related geometry. It does not mean every predictive embedding is literally equal to an SVD result, or that the same dimensions will align. Weighting, sampling, dimensionality, optimization, corpus composition, and regularization all affect the learned space.
Contextual language models add another distinction. A static word embedding gives one vector per vocabulary item. A contextual model can produce different token representations depending on surrounding text. A sentence embedding is a separate training target or pooling design intended to make whole-sentence comparison useful. Averaging token vectors is a baseline, not a guarantee of sentence-level semantics.
Inspecting a projection
The plot below is a mental model for PCA or SVD inspection. A projection compresses a high-dimensional space into two coordinates while retaining selected variance. Nearby points in the projection may be far apart in the full space, and points that overlap may still differ along discarded dimensions.
Use a projection to find obvious data issues: duplicate records, clusters dominated by one label, outliers, or a vocabulary split caused by formatting. Do not use a pretty cluster as evidence that the embedding understands the business concept. Test retrieval against labelled queries.
Real-world application: semantic search
A production semantic-search pipeline embeds documents and queries with a compatible, versioned model. It stores the vectors with document ids, source metadata, access-control attributes, and the exact preprocessing configuration. A query vector is compared with indexed vectors, often with an approximate-nearest-neighbor index when the collection is large. The application then applies permissions, freshness rules, and business filters before displaying results.
The vector index is not the whole system. A result can be geometrically close and still be unauthorized, stale, or missing the exact identifier the user needs. Hybrid retrieval can combine lexical matching for names and codes with vector similarity for paraphrases. Reranking can inspect a smaller candidate set with a more expressive model.
Store the embedding model name and version with every vector batch. When the model changes, dimensions, normalization, neighborhood structure, and score distributions can change. Re-embed and re-evaluate rather than appending vectors from incompatible spaces to one index. Keep an old index during migration so results can be compared and rollback remains possible.
Similarity is an instrument, not a verdict
Cosine similarity answers one narrow question: how aligned are two vectors after their lengths are divided out? It does not answer whether a document is authoritative, current, safe to disclose, or sufficient to answer a question. Treat the score as one signal in a retrieval policy. A useful policy may require a minimum score, a maximum age, a matching access scope, and at least one lexical confirmation for identifiers.
The threshold should come from evaluation rather than intuition. Build a small set of real queries with labelled relevant documents. Measure recall at the number of chunks you pass downstream, inspect false positives, and vary the threshold. If the application must answer only when evidence is strong, measure abstention coverage: how often the system answers, and how often those answers are supported. A higher threshold can improve precision while leaving more queries for search refinement or a human.
Sentence embeddings add another layer of training choice. A sentence encoder is optimized so whole-sentence representations support a task such as semantic similarity or retrieval. Mean-pooling contextual token vectors may be a useful baseline, but it is not automatically equivalent to a sentence encoder. Compare pooling methods on the task you actually have, and keep query and document instructions consistent when the model specifies them.
Finally, embedding privacy is easy to underestimate. A vector is not plain text, but it is derived from text and can support similarity searches against sensitive material. Apply the same access controls, retention policy, deletion process, and tenant isolation to indexes as to source documents. Deleting a document means deleting its vector, metadata, caches, and evaluation copies, not only removing its row from a primary table.
That operational detail also affects debugging. Keep a traceable document id, chunk boundary, preprocessing version, model version, and retrieval score for each result shown during an evaluation run. Do not retain raw customer text in ordinary logs merely to make a failed search easier to inspect. Use redacted fixtures and controlled access to the source record. A retrieval system is a data system with a model attached, so its correctness includes both relevance and stewardship.
Make those fields part of the retrieval contract from day one, everywhere, always, explicitly.
Failure modes and debugging
- Wrong space comparison. Never compare vectors from incompatible models without an alignment procedure.
- Normalization mismatch. Cosine ranking and inner-product ranking are equivalent only under the appropriate normalization.
- Data leakage. A document may retrieve itself because its test copy was indexed. Separate evaluation data before building the index.
- Chunking dominates quality. A good model cannot recover context that was split across unrelated chunks or buried in boilerplate.
- Access control after retrieval. Filtering only after showing results can leak titles or snippets. Apply authorization before response construction.
- Projection overinterpretation. PCA and UMAP are views, not ground truth.
Limitations / when not to use
Embeddings do not replace exact search for invoice numbers, account ids, version strings, or legal clauses where a character-level match matters. They also cannot guarantee factuality. A close vector means the model considers two representations related under its training objective; it does not mean the source is correct or the answer is supported.
The local experiment is far too small to evaluate a production embedding model. Its geometry is intentionally fragile. Real evaluation needs representative queries, relevance labels, access-control cases, multilingual samples where applicable, and metrics such as recall at k or mean reciprocal rank. Retrieval quality should be measured before generation quality.
Exercises
- Change the context window and compare the nearest neighbors.
- Remove PPMI and inspect how common words change the geometry.
- Add a held-out query set and compute recall at three.
- Compare lexical, cosine, and hybrid ranking on identifiers and paraphrases.
- Simulate a model-version migration and quantify how many top results change.
Next in this path
The next article is Next-Token Prediction: The Learning Objective Behind Language Models. Embeddings give discrete inputs a continuous space; next-token prediction shows how a language model learns to assign probability to the next item in a sequence.
Sources
The research dossier is what-embeddings-encode. The distributional framing is recorded by the ACL Wiki distributional-hypothesis entry. Count-based vector-space structure follows Turney and Pantel, From Frequency to Meaning. Predictive embeddings follow Mikolov et al., Efficient Estimation of Word Representations, and negative sampling follows Distributed Representations of Words and Phrases. The implicit-factorization connection is Levy and Goldberg, Neural Word Embedding as Implicit Matrix Factorization. Sentence-level comparison is discussed by Reimers and Gurevych, Sentence-BERT. SVD behavior is checked against the NumPy documentation, and large-scale indexing against Faiss.
Research and code were last verified on 2026-08-05. The local coordinates are an educational result from the bundled corpus, not a claim about general semantic quality.