I understand the backpropagation algorithm of neural networks, and how the error propagates backwards in layers. That is, I understand that given a 3-layer feed forward network, the amount to change W1 is dependent on the weights from layers 2 and 3, as well as the derivatives of their activation functions.

Question: When your first layer is an embedding layer (i.e., consider initializing the embedding matrix with glove), how does the network update that matrix using backpropagation? How do you represent that layer as an equation consisting of the input and some weight matrix?

  • $\begingroup$ Do you mean that you first learn vector embeddings from GloVe (lets say for words) and the use these as input to some another neural network designed for a different application (lets say translation) ? $\endgroup$
    – mausamsion
    Jun 13 '18 at 0:47
  • $\begingroup$ @flyingDope: yes, exactly. $\endgroup$
    – David
    Jun 13 '18 at 15:08

An embedding layer is in fact a linear layer. It maps the input, using a matrix multiplication, to the output, without any activation function after the multiplication. Therefore, the backpropagation is exactly as you would do with linear layer.

Why don't we just call it linear layer, then?

At theory level, an embedding layer performs a matrix multiplication to the input. However, in practice, the coded implementation is slightly different. This is due to the fact that the input, as a category, is incoded in a one-hot way, and the matrix multiplication by a one-hot encoded vector is as easy as a look-up, so there is no need to multiply the whole matrix.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.