Embeddings mathematically: what cosine similarity measures

Every explanation of embeddings stops at the same place: "they turn text into vectors that capture meaning, and similar things end up close together."

True, and useless the moment something goes wrong. Why is every pair of your documents 0.8 similar? Why did normalising change your results? Why does 1536 dimensions not beat 768? Those questions all have answers, and the answers are geometric.

An embedding is a point on a sphere

An embedding is a vector in R^d. For a 768-dimensional model, one document is 768 numbers. Nothing mystical — the interesting part is what the geometry of that space encodes.

"king" and "queen" have similar numbers in similar places. "bicycle" does not. That similarity in direction is the entire mechanism.

Cosine similarity is an angle, not a distance

The standard measure:

                    a · b            sum(a_i * b_i)
cos(a, b) = ----------------- = -----------------------------
              ||a|| * ||b||      sqrt(sum a_i^2) * sqrt(sum b_i^2)

import numpy as np


def cosine(a, b):
    return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))

The numerator is the dot product — how much the two vectors point the same way. The denominator divides out both magnitudes. What remains is purely the angle between them.

cos =  1.0   same direction        -->  0 degrees
cos =  0.0   perpendicular         --> 90 degrees
cos = -1.0   opposite directions   --> 180 degrees

Why angle and not distance? Because magnitude in an embedding space mostly encodes things you do not care about — text length, token frequency, how "confident" the encoder was. Two documents saying the same thing at different lengths point the same way but have different norms. Angle ignores that; Euclidean distance does not.

Normalising makes dot product and cosine the same thing

If ||a|| = ||b|| = 1 then the denominator is 1, and:

cos(a, b) = a · b

This is why every vector database tells you to L2-normalise before indexing. It turns an expensive-ish similarity into a raw dot product — which is a single matrix multiply over the whole corpus.

def l2_normalise(X):
    """X: (n, d). Returns unit vectors, safe against zero rows."""
    norms = np.linalg.norm(X, axis=1, keepdims=True)
    return X / np.maximum(norms, 1e-12)


V = l2_normalise(embeddings)          # (n, d), every row on the unit sphere
sims = V @ query                      # (n,) — all cosine similarities, one matmul

And it explains a bug people hit constantly: if you normalise at index time but not at query time, your scores are wrong — silently, and in a way that still returns plausible-looking results. Normalise both, always, in the same place in the code.

Euclidean distance on normalised vectors is cosine in disguise

Worth knowing, because it explains why the choice of metric often does not matter:

||a - b||^2 = ||a||^2 + ||b||^2 - 2(a · b)
            = 1 + 1 - 2 cos(a, b)          (unit vectors)
            = 2 - 2 cos(a, b)

Distance is a strictly decreasing function of cosine similarity. On normalised vectors, ranking by L2 distance and ranking by cosine give identical orderings. If your database offers both and your vectors are normalised, pick either.

Why high dimensions behave strangely

Intuition built in 2D and 3D actively misleads here.

Random vectors are almost always near-orthogonal. Draw two random unit vectors in R^d. Their expected cosine similarity is 0, with standard deviation roughly 1/sqrt(d).

for d in [2, 10, 100, 768, 3072]:
    A = l2_normalise(np.random.randn(2000, d))
    B = l2_normalise(np.random.randn(2000, d))
    sims = (A * B).sum(1)
    print(f"d={d:5d}  mean={sims.mean():+.4f}  std={sims.std():.4f}")

# d=    2  mean=+0.0031  std=0.7071
# d=   10  mean=-0.0018  std=0.3162
# d=  100  mean=+0.0009  std=0.1000
# d=  768  mean=-0.0002  std=0.0361
# d= 3072  mean=+0.0001  std=0.0180

That 1/sqrt(d) is the whole story of why embeddings work. In 768 dimensions there is enormous room for concepts to be distinguishable — unrelated things land essentially perpendicular, so any similarity above the noise floor is real signal.

It also tells you what a similarity score means. A cosine of 0.3 between random 768-dim vectors is roughly 8 standard deviations from the mean. That is not "somewhat related" — for random vectors it is essentially impossible. Which is exactly why real embedding similarities look so inflated, and why you should never compare thresholds across models with different dimensionality.

The anisotropy problem

Here is the thing that trips up most people building their first retrieval system.

Real embeddings are not spread evenly over the sphere. They occupy a narrow cone. Compute the pairwise similarity of a thousand unrelated documents from a real encoder and you will see a mean around 0.6–0.8, not 0.

Ideal (isotropic)                Reality (anisotropic)

        .  .  .                          ...
     .    .    .                        .....
   .   .  .  .   .                     .......      <- everything lives here
     .    .    .                        .....
        .  .  .                          ...

mean similarity ~ 0.0             mean similarity ~ 0.7

Cause: token frequency. Common tokens dominate the learned representation and drag every vector toward a shared mean direction. The relative ordering is still informative — that is why retrieval works at all — but the absolute numbers are compressed into a narrow band.

Practical consequences:

Absolute thresholds are meaningless across models. "Keep results above 0.75" might keep everything on one model and nothing on another. Threshold on rank, or calibrate the threshold empirically per model.

Centring helps. Subtract the corpus mean before comparing, and the distribution reopens:

mu = embeddings.mean(axis=0)
centred = l2_normalise(embeddings - mu)     # subtract, then re-normalise

This is the cheap version of "all-but-the-top" and whitening post-processing. It costs one vector and one subtraction, and it frequently sharpens the separation between related and unrelated pairs. Measure it on your own retrieval evaluation rather than adopting it blindly — it helps often, not always.

What dimensionality actually buys

More dimensions means more capacity to distinguish concepts. It also means more storage, more compute, and diminishing returns.

The reason returns diminish is that the intrinsic dimensionality of your data is much lower than the embedding dimension. A corpus of support tickets might genuinely vary along a few hundred meaningful axes. Extra dimensions beyond that encode noise.

You can measure this directly with PCA:

from sklearn.decomposition import PCA

pca = PCA().fit(embeddings)
cum = np.cumsum(pca.explained_variance_ratio_)
print("dims for 90% variance:", np.searchsorted(cum, 0.90) + 1)
print("dims for 99% variance:", np.searchsorted(cum, 0.99) + 1)

# dims for 90% variance: 187
# dims for 99% variance: 512

If 90% of the variance in your 768-dim embeddings lives in 187 dimensions, a 3072-dim model is buying you very little for four times the storage.

This is also why Matryoshka embeddings are useful: models trained so that the first k dimensions are themselves a valid embedding. You truncate to 256 dims for a fast first pass and use the full vector to rescore the shortlist — the same coarse-then-exact pattern that runs through the whole retrieval stack.

coarse = l2_normalise(embeddings[:, :256])       # cheap, 3x smaller
shortlist = np.argpartition(-(coarse @ q[:256]), 200)[:200]
exact = embeddings[shortlist] @ q                # rescore with all 768
top10 = shortlist[np.argsort(-exact)[:10]]

Why word arithmetic mostly does not work any more

king - man + woman ≈ queen is the famous demonstration, and it belongs to static embeddings — word2vec, GloVe — where every word had exactly one vector.

Modern sentence embeddings are contextual: "bank" in a river sentence and "bank" in a finance sentence get different vectors. That is a large improvement for retrieval and it breaks the arithmetic, because there is no single stable "king" vector to subtract from.

There is still directional structure in the space — sentiment, formality and language often have consistent directions you can find with a bit of probing — but the clean analogy demos do not transfer. If you read a tutorial doing vector arithmetic on sentence embeddings, it is showing you a coincidence.

Sanity checks worth running on any new model

Before you trust an embedding model on your data, five minutes of measurement:

def diagnose(embeddings, labels=None, sample=2000):
    X = l2_normalise(embeddings[:sample])
    S = X @ X.T
    off = S[~np.eye(len(S), dtype=bool)]

    print(f"pairwise mean : {off.mean():.3f}   (near 0 = isotropic)")
    print(f"pairwise std  : {off.std():.3f}    (bigger = more separation)")
    print(f"p99 similarity: {np.percentile(off, 99):.3f}")

    if labels is not None:
        same = np.array(labels)[:, None] == np.array(labels)[None, :]
        same &= ~np.eye(len(S), dtype=bool)
        print(f"same-label    : {S[same].mean():.3f}")
        print(f"diff-label    : {S[~same & ~np.eye(len(S),dtype=bool)].mean():.3f}")
        print(f"separation    : {S[same].mean() - S[~same].mean():.3f}")

The number that matters is separation — the gap between same-label and different-label similarity. A model with mean similarity 0.8 and separation 0.15 will outperform one with mean 0.2 and separation 0.05, even though the second looks better isotropically. Absolute similarity is cosmetic; the gap is what retrieval actually uses.

Frequently asked questions

Should I normalise my embeddings?

Yes, and consistently. Normalising makes dot product equal cosine similarity, which is faster and is what most vector indexes assume. The dangerous case is normalising at index time but not at query time — the scores are then wrong in a way that still returns plausible results.

Why are all my similarity scores between 0.7 and 0.9?

Anisotropy. Real embedding spaces occupy a narrow cone rather than the full sphere, because frequent tokens pull every vector toward a shared direction. The ordering is still meaningful; the absolute values are compressed. Threshold on rank, not on a hard cosine cutoff, and try subtracting the corpus mean before comparing.

Is a higher-dimensional embedding model better?

Usually only slightly, and it always costs more. Run PCA on your embeddings: if 90% of the variance sits in 200 dimensions, a 3072-dim model is mostly storing noise. Measure retrieval quality on your own evaluation set before paying for the bigger vector.

Cosine similarity or Euclidean distance?

On normalised vectors they produce identical rankings, because ||a-b||^2 = 2 - 2cos(a,b). On unnormalised vectors, cosine is usually what you want, since magnitude mostly encodes length and frequency rather than meaning.

Why does king - man + woman not work with my model?

That demonstration comes from static word embeddings, where each word had one fixed vector. Modern sentence embeddings are contextual, so there is no single stable vector per word and the arithmetic no longer holds.

Next reading on Sythra Articles

• Vector databases under the hood
• Building RAG properly
• The attention mechanism, worked out by hand