I am observing my word2vec model learning context words as most similar rather than words in similar contexts. I don't understand why it (word2vec in general, not my model in particular) can behave like that and would like to know why.
I have implemented the original word2vec in keras. I chose the variant with the dot product layer rather than the hierarchical softmax and trained the model on a Wikipedia dump that I split into 5-grams. For each word I construct 8 pairs with a binary target label as training items. I use the 4 context words with the label True and choose 4 random words that are not one of the context words with the label 0.
Intuitively this model should learns similar representations for words in similar context, because it modifies the representation of these words in a similar manner, as it optimizes them independently with similar context words. So these similar words are not directly made similar but rather indirectly, as they are subject to similar nudges due to their similar contexts.
The model is this:
input_target = Input((1,)) input_context = Input((1,)) embedding = Embedding(vocab_size, vector_dim, input_length=1, name='embedding') target = embedding(input_target) target = Reshape((vector_dim, 1))(target) context = embedding(input_context) context = Reshape((vector_dim, 1))(context) dot_product = Dot(axes=1)([target, context]) dot_product = Reshape((1,))(dot_product) output = Dense(1, activation='sigmoid')(dot_product) model = Model(inputs=[input_target, input_context], outputs=output) model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
However, what I observe from training this model is, that when I rank the most similar words given a certain word, that I get words that appear in the context of that word, rather than words that appear in similar context, as the most similar words.
words most similar to "plant": rank | word 0 | amount 1 | surface 2 | electron 3 | mass 4 | plant # also: Why is plant not most similar to plant? How can that happen? 5 | fluid 6 | air 7 | metal 8 | molecule 9 | cell 10 | electric 11 | per 12 | oxygen 13 | demonstrate 14 | smooth
To me that looks a lot more like words that appear in the context of plant rather than words that appear in similar context as it.
The function to compute these is:
def get_most_similar(word_vector, embeddings, n=15): """ find the `n` words that are most similar to `word_vector` in `embeddings` measured by their cosine similarity """ v = word_vector m = embeddings cosines = (np.dot(v, m.T))/(np.linalg.norm(m.T, axis=0)*np.linalg.norm(v)) ranked_by_similarity = np.argpartition(cosines, -n)[-n:] return reversed(ranked_by_similarity)
Is there a simple reason for that?
I have the following other parameters:
word vector size: 300 batch size: 128 vocabulary size: 169161 (distinct lemmas) training sample count: 27793586 (5-grams, overlapping within sentences)
I trained the model for 1 epoch and did only observe marginal further convergence into the second epoch.