1
$\begingroup$

I'm studying Word2Vec algorithm, and so far i understood that, in the case of input context bigger than 1 (so multiple words) we have our hidden layer that performs averaging between the inputs (as explained here: Word2Vec CBOW)

I didn't understand how does backpropagation work in this case.

When i start backpropagating from the softmax output layer and i reach for the hidden layer in my process, am i coping the weight over all the "same position" inputs?

To make my question clearer, if i have 2 context words coded as one hot vectors and i call the first weight of the first word as W11 and the first weight of the second word as W21, when i'm backpropagating, those two weights will be updated with se same value, right?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Highly unlikely.

Simple thought experiment. If the vertices (graphically == weights) are the same for these two nodes (first cell of the vector for the first word and the first cell of the vector for the second word in context) (what I am trying to say they are fully connected) BUT input values when you feed-fowarded these values were different, not only that this difference spaned accros other layers, hence you can not expect that weight updates via backprop will be same.

Improve this answer
$\endgroup$
1
  • $\begingroup$ So how does the backpropagation work in the case of an averaging layer? In the forward pass the output of my hidden layer (the one mentioned in the link) is the average of all the inputs received from the single hidden neuron. What happens when i start backpropagating and i have to update all those weights which were averaged? $\endgroup$
    – Mattia Surricchio
    Commented Dec 23, 2019 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.