0
$\begingroup$

Could you please explain how do we get output layer in this architecture (vectors [0.2, 0.8, -1.4, 1.2] and [-0.3, 0.2, -0.7, 0.1]). I understand that layer before are embeddings of word "brown". But how do we get vector [0.2, 0.8, -1.4, 1.2]? I thought it should be dot product of "brown"'s embeddings with "quick"'s embeddings? Could you please describe in details how this part works?

enter image description here

$\endgroup$
2

1 Answer 1

0
$\begingroup$

We have 2 embedding matrices(U,V) that are learnt during word2vec training. U has shape (vocab_size, dimensions) V has shape (dimension, vocab_size) For any given pair of target, context:

  1. We pick embedding of target from U(one of the rows in U matrix) and calculate it"s dot product with all the vectors in V matrix leading to a score vector(third from last vector). score_vector has dimension vocab_size*1

This score vector(output vector in question) is taken into probability space by using softmax operation. Now we have predicted probabilities ie softmax(score_vector) and actual probability vector ie one-hot vector of context word.

In other words we want probability distribution that predicts context word(here quick) with higher probability than the rest of the words.

Once we have predicted vector and actual vector we calculate loss(ie. cross entropy) and backpropogate it to modify both U and V.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.