# Word2Vec: Why do some dimensions of an embedding have an interpretation, and why does addition/subtraction of embedding vectors work?

I'm reading about Word2Vec from this source: http://jalammar.github.io/illustrated-word2vec/. Below is the heatmap of the embeddings for various words. In the source, it's claimed that we can get an idea on what the different dimensions "mean" (their interpretation) based on their values for different words. For example, there's a column that's dark blue for every word except WATER, so that dimension may have something to do with the word representing a person.

Secondly, there's a famous example that "king" - "man" + "woman" ~= "queen", where the word in quotation means the embedding of that word.

My questions are:

1. I don't quite understand the mechanism as to how any dimension of an embedding goes on to have a tangible, interpretable meaning. I mean, the individual components of embedding vectors could've very well been completely arbitrary devoid of meaning, and the whole embedding approach still could've worked in that scenario, since we're interested in the vector as a whole. Is there an online explanation or a paper that I can look at to understand this phenomenon?
2. Why does this addition/subtraction of vectors to give the relevant embedding vector for "queen" work so nicely? In one source, the explanation is given as follows:

This works because the way that the neural network ended up learning about related frequencies of terms ended up getting encoded into the W2V matrix. Analogous relationships like the differences in relative occurrences of Man and Woman end up matching the relative occurrences of King and Queen in certain ways that the W2V captures.

This seems like a broad, vague kind of an explanation. Is there any online resource or paper that explains (or better yet, proves) why this property of embedding vectors should hold?

• Dec 24, 2020 at 16:21
• @BenReiniger: Thank you! That's super helpful Dec 24, 2020 at 17:41
• I suspect that 1 is mostly a coincidence. There are trillions of concepts you might think of as defining some linear scale, millions of which are partially captured by the embedding space, and some of those are bound to (completely by chance) lie principally along one of the dimensions of the embedding. (But that's all speculation, so I'll leave this as a comment.) Dec 26, 2020 at 4:41
• @BenReiniger: Yeah there's a lot of follow-up questions one could ask related to 1. For example if we change the dimension from 300 to something else, would concepts be still captured in individual dimensions? I guess one could answer this through explicit experimentation if one has bandwidth Dec 26, 2020 at 8:53
• One possible explanation for 1. (useful for 2. as well) is that word2vec and similar tools extract the features through a process which checks inter-relations between words in a corpus (eg how many times this word is close to that word). But exactly these same inter-relations is what expresses meaning in natural language (eg natural language grammars). For example "water" would be distant on average from concepts/words such as "king" or "queen" and so on.. On the other hand "water" would be close on averege to "thirsty", "wet" and so on.. Dec 26, 2020 at 17:57

TL;DR: A theoretical/mathematical explanation for why word2vec/GloVe embeddings of analogies appear to form parallelograms, and so can be "solved" by adding/subtracting embeddings, is given here, as summarised in this blog. More explanation of w2v is given here.

The dimensions of word2vec (or GloVe, etc) word embeddings are not directly interpretable, but capture correlations in word statistics, which reflect meaningful semantics (e.g. similarity), so some dimensions may happen to be interpretable.

The embedding of a word is effectively a low-rank projection of the co-occurrence statistics of that word with all other words (like what you would get from PCA/SVD - but that would require an unweighted least square loss function).

That projection in word2vec is probability weighted and non-linear, making it difficult to interpret what any dimension "means". Also, if the embedding matrix $$W$$ (all embeddings $$w_i$$ stacked together) is rotated by any rotation matrix $$R$$, and $$R^{-1}$$ applied to the other embedding matrix $$C$$, the transformed embeddings perform identically. So there isn't a unique solution, but an equivalence class of solutions, meaning the values in embeddings aren't necessarily meaningful in their own right, only when considered relative to each other.

The theoretical explanation of analogies is too long to summarise here, but it boils down to word embeddings capturing log probabilities, so adding embeddings is equivalent to multiplying probabilities and so is meaningful. I gather it's bad form to include link explanations, but the two linked research papers should last in perpetuity.

• Thank you! Your arxiv paper seems super interesting, and I will go through it in as much detail as I can and revert back with follow-up questions if that's okay with you. Jul 9, 2021 at 11:39
• Sure, no problem (but not sure which paper you mean, one is ICML, other NeurIPS).
– Carl
Jul 9, 2021 at 12:13
• I'll start with the ICML one, then look at NeurIPS Jul 9, 2021 at 12:16
• I'd recommend the blog as the quickest intro, but entirely up to you.
– Carl
Jul 9, 2021 at 12:18

The dimensions for an embedding space are only be accidentally interpretable.

However, vectors through the space can be interpretable. That is why word analogies are possible in an embedding space . The addition / subtraction of word vectors describe another vector through the embedding space. For example, "king" - "man" + "woman" approximate the "queen" vector.

Words are consistently used in relationship to other words. Word embeddings can model these consistent relationship by finding which words co-occur together and projection the words into lower dimension space that retains the most common occurrences.