Possible reasons for word2vec learning context words as most similar rather than words in similar contexts

Question

I am observing my word2vec model learning context words as most similar rather than words in similar contexts. I don't understand why it (word2vec in general, not my model in particular) can behave like that and would like to know why.

I have implemented the original word2vec in keras. I chose the variant with the dot product layer rather than the hierarchical softmax and trained the model on a Wikipedia dump that I split into 5-grams. For each word I construct 8 pairs with a binary target label as training items. I use the 4 context words with the label True and choose 4 random words that are not one of the context words with the label 0.

Intuitively this model should learns similar representations for words in similar context, because it modifies the representation of these words in a similar manner, as it optimizes them independently with similar context words. So these similar words are not directly made similar but rather indirectly, as they are subject to similar nudges due to their similar contexts.

The model is this:

input_target = Input((1,))
input_context = Input((1,))

embedding = Embedding(vocab_size, vector_dim, input_length=1, name='embedding')

target = embedding(input_target)
target = Reshape((vector_dim, 1))(target)
context = embedding(input_context)
context = Reshape((vector_dim, 1))(context)

dot_product = Dot(axes=1)([target, context])
dot_product = Reshape((1,))(dot_product)

output = Dense(1, activation='sigmoid')(dot_product)

model = Model(inputs=[input_target, input_context], outputs=output)
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

However, what I observe from training this model is, that when I rank the most similar words given a certain word, that I get words that appear in the context of that word, rather than words that appear in similar context, as the most similar words.

For example:

words most similar to "plant":

rank |   word

0    |   amount
1    |   surface
2    |   electron
3    |   mass
4    |   plant # also: Why is plant not most similar to plant? How can that happen?
5    |   fluid
6    |   air
7    |   metal
8    |   molecule
9    |   cell
10   |   electric
11   |   per
12   |   oxygen
13   |   demonstrate
14   |   smooth

To me that looks a lot more like words that appear in the context of plant rather than words that appear in similar context as it.

The function to compute these is:

def get_most_similar(word_vector, embeddings, n=15):
    """
    find the `n` words that are most similar to `word_vector` in `embeddings`
    measured by their cosine similarity
    """ 
    v = word_vector
    m = embeddings
    cosines = (np.dot(v, m.T))/(np.linalg.norm(m.T, axis=0)*np.linalg.norm(v))
    ranked_by_similarity = np.argpartition(cosines, -n)[-n:]
    return reversed(ranked_by_similarity)

Is there a simple reason for that?

I have the following other parameters:

word vector size: 300
batch size: 128
vocabulary size: 169161 (distinct lemmas)
training sample count: 27793586 (5-grams, overlapping within sentences)

I trained the model for 1 epoch and did only observe marginal further convergence into the second epoch.

Answer 1

Nice example of using embeddings with Keras.

If I interpret it correctly, there is a big difference between your implementation and the "original word2vec". The original framework operates not with one 'embeddings' vector_size x vector_dim weight matrix as in your case, but with two matrices (or layers): the "projection layer" which maps the input to a vector of dimension vector_dim and the "hidden layer" which maps this vector to probabilities over all vocabulary words.

The hidden, or prediction layer, is often discarded after the training, although it can be useful.

If your code indeed forces those layers to share weights, then we get something also interesting, but different, it is plausible the embeddings reflect co-ocurrences - see this question which asks specifically what happens if the two layers are shared.

Either way 'plant' should be most similar to 'plant' for cosine similarity - or there are some vectors that are exactly the same (normalized).

According to the original word2vec paper, hierarchical softmax is used to speed up the training. So if you don't use it and have two weight layers, your version might become too slow. I can recommend for example the gensim library instead that implements the soft-max and should run on a modern laptop in a manageable time with your setup.

Answer 2

Neural networks are different from other Machine Learning techniques in that they have to learn by repetition. In Neural Network parlance, the repetition is called an epoch. During an epoch, each instance, in your case each word, is evaluated and the error is applied backwards through the layers and the weights are adjusted. The changes to the weights can happen more or less aggressively based on the hyperparameters, but they tend to only move a little at a time.

You have trained your word2vec model for two epochs. I didn't see how you initialized the weights for your model, but typically they are set to random values or set to a zeros. Given your results look a little random, I am sure it is because the model has barely begun to learn.

In order to get a well trained word embedding model, or any other Neural Network, you may need to train for hundreds, thousands, or even millions of epochs. This is why the pre-trained word embedding models are so valuable even though they are freely available. If you just want a well trained embedding model, download it. If you need to incorporate some domain specific usage, you can further train the downloaded embeddings.

HTH