I'm trying to perform sentiment analysis on some data using keras.I'm using embedding layer and then LSTM. I know that embedding layer decreases the sparsity of the one hot encodings of the words and its parameters are trained while back-propagation, but I don't know the mathematics of its implementation.

Thanks in advance.


Like all hidden layers in a neural network, an embedding layer can be thought of as a feature extractor with parameters that are automatically learned during training. So, like any other layer, the parameters are adjusted during training by the backpropagation algorithm.

The specific equations used to compute error gradients and update weights are likely to depend on the optimizer used to train the network. The classic optimizer is stochastic gradient descent, but more sophisticated optimizers like Adagrad, RMSProp, and ADAM are commonly used nowadays.

  • 1
    $\begingroup$ Thankyou Zach!, I went through the sequence models course by AndrewNg on coursera, He explains it by talking about an Embedding matrix which is used to generate embeddings. The parameter of the embedding layer is that embedding matrix and the elements of the matrix are learnt during training by the backpropagation algorithm as you mentioned. $\endgroup$ Jun 6 '19 at 4:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.