I've trained a simple skipgram NNLM (window size = 5) on The Hobbit. This is the rough pseudocode:
for e in epochs: for i in idx: loss_batch = 0 for b in batch_size: input = data[i+b] output = nnlm(input) loss_window = neg_log_lhd(output, target) loss_batch += loss_window loss_batch /=batch_size loss_batch.backward()
idx here is the index of the first input word in the batch. Loss is accumulated over the whole batch (if
batch_size=128, it means there are 128 sliding windows for which the loss is calculated) and backpropagated before moving to the next one.
The model seems to converge, judging by the curve of the loss function, but I'm concerned about the vector representation of words. I computed cosine similarity for 4 different words: 'bilbo','baggins', 'gandalf', 'thorin'. I expected the similarity between the first two to be very high, as they are used interchangeably, or very frequently together. Nevertheless, it starts with
-0.042 and is
-0.04 after 40 epochs (~400 batches/epoch). Obviously something's wrong here.
What could possibly be the reason for the situation when the model converges, but the distance between word embeddings doesn't change?