How do we get output layer in skip-gram?

Question

Could you please explain how do we get output layer in this architecture (vectors [0.2, 0.8, -1.4, 1.2] and [-0.3, 0.2, -0.7, 0.1]). I understand that layer before are embeddings of word "brown". But how do we get vector [0.2, 0.8, -1.4, 1.2]? I thought it should be dot product of "brown"'s embeddings with "quick"'s embeddings? Could you please describe in details how this part works?

Answer 1

We have 2 embedding matrices(U,V) that are learnt during word2vec training. U has shape (vocab_size, dimensions) V has shape (dimension, vocab_size) For any given pair of target, context:

1. We pick embedding of target from U(one of the rows in U matrix) and calculate it"s dot product with all the vectors in V matrix leading to a score vector(third from last vector). score_vector has dimension vocab_size*1

This score vector(output vector in question) is taken into probability space by using softmax operation. Now we have predicted probabilities ie softmax(score_vector) and actual probability vector ie one-hot vector of context word.

In other words we want probability distribution that predicts context word(here quick) with higher probability than the rest of the words.

Once we have predicted vector and actual vector we calculate loss(ie. cross entropy) and backpropogate it to modify both U and V.