Why Embeddings Produce Similarity

Embedding spaces preserve similarity because training preserves context: words that share contexts get pushed toward each other by the objective.

The distributional hypothesis is the founding assumption: words that appear in similar contexts tend to carry similar meaning. Embedding models operationalize it. A word that appears in the same neighborhoods as another word receives a vector close to that vector, and cosine similarity between the two vectors measures the overlap.

This note derives the geometry from a co-occurrence matrix and a ten-line training objective, then shows the three pressure points — direction versus magnitude, antonyms, and polysemy — each followed by the fix that downstream systems apply.

The naïve version: count the neighbours#

The simplest mechanism that satisfies the hypothesis counts co-occurrences directly. Build a vocabulary, choose a window size, and record how often each word appears near each other word:

from collections import Counter, defaultdict
 
def build_cooccurrence(sentences, vocab, window=2):
    index = {w: i for i, w in enumerate(vocab)}
    counts = defaultdict(Counter)
    for sentence in sentences:
        tokens = [t for t in sentence.split() if t in index]
        for i, token in enumerate(tokens):
            for j in range(max(0, i - window), min(len(tokens), i + window + 1)):
                if i == j:
                    continue
                counts[token][tokens[j]] += 1
    return counts
 
corpus = [
    "doctor treats patient in hospital",
    "nurse treats patient in hospital",
    "doctor checks pulse in hospital",
    "hot water steams in kitchen",
]
vocab = ["doctor", "nurse", "patient", "hospital", "hot", "water", "kitchen"]
counts = build_cooccurrence(corpus, vocab, window=2)
for word in vocab:
    print(word, dict(counts[word]))

The matrix that falls out of this code has rows that are context distributions. Two words with similar rows share most of their context counts, and a similarity function over rows recovers the hypothesis numerically. The problem is dimensionality: a corpus with $V$ vocabulary items produces a $V \times V$ matrix, almost all of it zeros, and pairwise comparisons over raw counts are dominated by the most frequent contexts.

First principles: from counts to vectors#

Matrix factorisation compresses the counts while preserving the geometry. The exact procedure that the predictive models approximate is singular value decomposition on a transformed co-occurrence matrix:

import math
 
def ppmi(counts_word, counts_context, total):
    """Positive pointwise mutual information for one (word, context) pair."""
    pmi = math.log((counts_word + 1e-9) * total /
                   ((counts_context + 1e-9) * (total + 1e-9) + 1e-9))
    return max(0.0, pmi)
 
def make_ppmi_matrix(counts, vocab):
    total = sum(sum(c.values()) for c in counts.values())
    context_totals = {w: 0 for w in vocab}
    for word in vocab:
        for ctx, c in counts[word].items():
            context_totals[ctx] += c
    matrix = []
    for word in vocab:
        row = [ppmi(counts[word].get(ctx, 0), context_totals[ctx], total)
               for ctx in vocab]
        matrix.append(row)
    return matrix

Raw counts are replaced by positive pointwise mutual information, which measures how much more often a pair co-occurs than chance would predict. The transformation downweights the function words that appear everywhere, and each row now encodes informative contexts rather than frequent ones. Word vectors are then the leading singular vectors of this matrix, and cosine similarity over those vectors measures overlap of informative context.

Predictive training produces the same geometry without ever forming the matrix. The skip-gram objective asks the model to predict a word's contexts from the word itself:

import math
import random
 
def sigmoid(x):
    return 1.0 / (1.0 + math.exp(-x))
 
def dot(a, b):
    return sum(x * y for x, y in zip(a, b))
 
def train_skipgram(pairs, dim=32, lr=0.05, epochs=5, seed=0):
    random.seed(seed)
    vocab = sorted({w for pair in pairs for w in pair})
    idx = {w: i for i, w in enumerate(vocab)}
    dim_vecs = [[random.uniform(-0.5, 0.5) for _ in range(dim)]
                for _ in vocab]
    context_vecs = [[random.uniform(-0.5, 0.5) for _ in range(dim)]
                    for _ in vocab]
    negatives = vocab
    for _ in range(epochs):
        for word, context in pairs:
            w = idx[word]
            target = idx[context]
            # positive pair
            score = dot(dim_vecs[w], context_vecs[target])
            grad = (sigmoid(score) - 1.0) * lr
            for k in range(dim):
                dim_vecs[w][k] += grad * context_vecs[target][k]
                context_vecs[target][k] += grad * dim_vecs[w][k]
            # negative samples, typically 5-15 per positive pair
            for _ in range(5):
                neg = random.choice(negatives)
                n = idx[neg]
                score = dot(dim_vecs[w], context_vecs[n])
                grad = sigmoid(score) * lr
                for k in range(dim):
                    dim_vecs[w][k] += grad * context_vecs[n][k]
                    context_vecs[n][k] += grad * dim_vecs[w][k]
    return {word: dim_vecs[idx[word]] for word in vocab}

Bottleneck one: the matrix is enormous, the projection is lossy#

A corpus with $V = 50{,}000$ vocabulary items produces a co-occurrence matrix with $2.5$ billion entries. Direct factorisation is infeasible; the predictive objective is the practical alternative. Negative sampling replaces the softmax over the full vocabulary with a small set of noise words, turning each update into a handful of dot products.

def negative_sampled_update(vecs, context_vecs, pair, negatives, lr):
    word, context = pair
    w_id = word
    updates = []
    score = dot(vecs[w_id], context_vecs[context])
    updates.append((context, sigmoid(score) - 1.0))
    for neg in negatives:
        score = dot(vecs[w_id], context_vecs[neg])
        updates.append((neg, sigmoid(score)))
    for ctx_id, grad in updates:
        g = grad * lr
        for k in range(len(vecs[w_id])):
            old = vecs[w_id][k]
            vecs[w_id][k] += g * context_vecs[ctx_id][k]
            context_vecs[ctx_id][k] += g * old
    return updates

Five to fifteen negatives per positive pair is the standard ratio. Each update touches only the word's own vector plus the sampled contexts, so the cost is independent of vocabulary size. The learned vectors are mathematically equivalent to factorising the shifted PMI matrix, with the shift determined by the negative-sampling rate.

Bottleneck two: magnitude carries frequency, not meaning#

The norm of a learned vector encodes how strongly the word is pulled toward its most common contexts. Frequent words and words in narrow technical domains grow large norms; rare words stay short. Cosine similarity removes this by measuring angle alone:

import math
 
def cosine(a, b):
    num = sum(x * y for x, y in zip(a, b))
    den_a = math.sqrt(sum(x * x for x in a))
    den_b = math.sqrt(sum(y * y for y in b))
    return num / (den_a * den_b + 1e-12)
 
def nearest(vector, table, top_k=5):
    scored = sorted(((cosine(vector, vec), word)
                     for word, vec in table.items()),
                    reverse=True)
    return scored[:top_k]

Direction is the informative part because the training objective pushes context-shared words toward common directions. Two words with similar context distributions share most of their directional components, and their vectors point in similar directions. The norm, which tracks frequency and informativeness, is discarded along with the training artefacts that produce it.

Linear analogies work for the same reason. If "king" and "man" differ in a direction that isolates a gender component, and "queen" differs from "woman" in the same direction, then translation by that vector moves along the semantic axis. The geometry is linear because the underlying co-occurrence statistics are linear, and the vectors are a compressed representation of them.

def analogy(a, b, c, table):
    """a : b :: c : ?  ->  table[a] - table[b] + table[c] nearest neighbour."""
    target = [x - y + z for x, y, z in zip(table[a], table[b], table[c])]
    candidates = {w: v for w, v in table.items() if w not in (a, b, c)}
    return nearest(target, candidates, top_k=3)
 
# king : man :: queen : woman
print(analogy("king", "man", "queen", vectors))

Bottleneck three: antonyms#

Antonyms break the intuition that closeness means similarity of meaning. "Hot" and "cold" land near each other, and that is not a bug in the embedding. They share contexts: both modify temperature nouns, appear in the same sentence frames, and are interchangeable in most positions.

pairs = [("hot", "cold"), ("good", "bad"), ("fast", "slow")]
for a, b in pairs:
    print(a, b, cosine(vectors[a], vectors[b]))

The embedding encodes substitutability, not polarity. Antonyms are found in nearly identical distributions, differing only in which direction along one axis they point. Their vectors are close because their contexts are nearly identical. The antonym relationship is real, and it lives in the geometry — as a direction within the near neighbourhood, not as a distance apart.

Any downstream use that treats proximity as semantic agreement will mistake "hot" for "cold". The embedding is correct about its own metric. The error is in reading the metric as meaning rather than as distributional overlap.

def antonym_direction(antonym_pair, table):
    """The polarity axis: cold - hot, averaged over pairs."""
    diffs = []
    for a, b in antonym_pair:
        diffs.append([y - x for x, y in zip(table[a], table[b])])
    n = len(diffs[0])
    mean = [sum(d[i] for d in diffs) / len(diffs) for i in range(n)]
    norm = math.sqrt(sum(x * x for x in mean))
    return [x / norm for x in mean]
 
def polarity(word, table, axis):
    return dot(table[word], axis)

Polarity is recoverable as a direction. A system that needs to distinguish "hot" from "cold" should project onto the antonym axis, not threshold the cosine.

Bottleneck four: polysemy#

Polysemy poses the opposite problem. "Bank" as a river edge and "bank" as a financial institution share a spelling, and a single word vector must compress both into one position. The result is a vector that sits between the two senses, similar to both and identical to neither.

Subword tokenization makes the problem harder, because "bank" as a token may also serve as a substring of "bankrupt" or "riverbank", and the token's embedding absorbs usage from all of them. A static embedding carries the average of every usage the corpus contained.

def sense_probe(word, contexts, table):
    """Compare the static vector against each usage context."""
    base = table[word]
    for ctx in contexts:
        print(ctx, cosine(base, table[ctx]))

The fix is architectural, not statistical: replace a single vector per word with a vector per occurrence.

Fix: contextual vectors#

Contextual models produce a different vector for each occurrence of the same word, conditioned on the surrounding sentence:

def contextual_vector(word, sentence, model):
    """Model runs the full sentence and reads out the word representation."""
    representations = model(sentence)
    position = sentence.split().index(word)
    return representations[position]
 
# "bank" in "the river bank" and "the bank raised rates"
v_river = contextual_vector("bank", "the river bank", model)
v_finance = contextual_vector("bank", "the bank raised rates", model)
print(cosine(v_river, v_finance))  # small: distinct senses, distinct vectors

Each occurrence is embedded by an encoder that sees the whole sentence, so the output vector is a function of word and context jointly. The geometry that was a single averaged point becomes a manifold of points per word type, with the senses forming separate regions. Polysemy stops being a collision and becomes a structure.

The tradeoff is cost: a static table is one lookup, while a contextual vector requires a full model pass per sentence. Production systems cache the static table for retrieval and invoke the contextual model where ambiguity matters.

The map is the corpus#

An embedding space is a compressed co-occurrence matrix, and its similarity structure is the structure of usage in the corpus the vectors were trained on. Change the corpus and the geometry moves. Train on biomedical text and "cold" lands near "fibrillation", not near "winter".

def top_neighbours(word, table, top_k=5):
    return nearest(table[word], table, top_k=top_k)
 
print(top_neighbours("cold", biomedical_vectors))
print(top_neighbours("cold", news_vectors))

The geometry is produced by a real mechanism, stated as a hypothesis in the 1950s and operationalized ever since. The mechanism preserves co-occurrence faithfully. Whether co-occurrence is meaning is a separate question, and the antonyms and polysemies are the evidence that it is close, not identical.

What to measure#

Similarity should be evaluated against ground truth derived from the same distributional assumption, and held-out prediction is the honest number:

def recall_at_k(queries, table, gold, k=10):
    hits = 0
    for query, expected in gold:
        neighbours = nearest(table[query], table, top_k=k)
        found = {w for _, w in neighbours}
        if expected in found:
            hits += 1
    return hits / len(gold)
 
def norm_frequency_correlation(table, frequencies):
    norms = [math.sqrt(sum(x * x for x in table[w])) for w in table]
    freqs = [math.log(1 + frequencies[w]) for w in table]
    # Pearson correlation between vector norm and log frequency
    mean_n = sum(norms) / len(norms)
    mean_f = sum(freqs) / len(freqs)
    cov = sum((n - mean_n) * (f - mean_f) for n, f in zip(norms, freqs))
    var_n = sum((n - mean_n) ** 2 for n in norms)
    var_f = sum((f - mean_f) ** 2 for f in freqs)
    return cov / math.sqrt(var_n * var_f)

The checklist for any embedding system:

• Test antonym pairs explicitly. A near-1.0 cosine for "hot"/"cold" is expected from this training objective; it confirms the geometry is working, not failing.
• Report norm-frequency correlation. A strong correlation means magnitude encodes frequency, and cosine should be used in place of dot product for retrieval.
• Evaluate on a held-out corpus from the same distribution. Similarity learned in-domain does not transfer across domains; the geometry is a property of the corpus.
• Probe polysemous words with usage contexts. A static vector cannot separate senses, and any task that requires sense separation needs a contextual model.
• Measure analogy accuracy separately from neighbour accuracy. Analogies depend on linear structure in the space, and a space can rank neighbours well while failing arithmetic.

An embedding is a compressed record of where words appeared together. The similarity in the space is the co-occurrence in the corpus, faithful and unedited. That is the grounding and the limit at once.