9
$\begingroup$

How is the embedding layer trained in Keras Embedding layer? (say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext)

Assume we do not use a pretrained embedding.

$\endgroup$
8
$\begingroup$

Both the answers are wrong.

An embedding layer is a trainable layer that contains 1 embedding matrix, which is two dimensional, in one axis the number of unique values the categorical input can take (for example 26 in the case of lower case alphabet) and on the other axis the dimensionality of your embedding space. The role of the embedding layer is to map a category into a dense space in a way that is useful for the task at hand, at least in a supervised task. This usually means there is some semantic value in the embedding vectors and categories that are close in this space will be close in meaning for the task.

This is related to one-hot encoding in a sense that it maps a discrete category into a vector feature representation. You could still do this for a neural network but if you use this in a dense layer you would create an enormous amount of weights of which most of them are not used regularly. Putting an embedding layer in between reduces the amount of learnable weights before feeding them to interact with other parts of your input. Another advantage is that the embedding matrix basically works as a lookup table, so you can really use the sparsity of the index of your category to look up what the current value of the embedding is and when applying backpropagating only adapting that entry of the weight matrix.

This blog post explains clearly about How the embedding layer is trained in Keras Embedding layer. Hope this helps.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Finally a good answer, thanks. We could add that Word2Vec is trained as a language modelling objective which makes it different from embeddings trained for a specific task as you said. $\endgroup$ – Elliot Jan 22 '19 at 9:33
0
$\begingroup$

As far as I understand, it is a simple autoencoder, meaning that all it does is trying to map the input into another space, so no fancy training, just some plain feed-forward and backprop. This is why it's rather fast to train.

If you want to use pre-trained embeddings, you can do it that way

| improve this answer | |
$\endgroup$
  • $\begingroup$ An autoencoder is a model which predicts itself. Is that what you meant? $\endgroup$ – kbrose Apr 2 '18 at 14:14
  • $\begingroup$ Indeed, so you can view it as a neural net with a single hidden layer, and where the input and output are identical I am not certain it is what Keras uses, but given the speed and the documentation, I'd tend to think that way $\endgroup$ – Valentin Calomme Apr 2 '18 at 21:23
  • $\begingroup$ The embedding layer itself does not do that. It only assigns a vector to each unique integer input. You could build an auto-encoding model around that to train the embeddings, but it is not part of the embedding layer. $\endgroup$ – kbrose Apr 3 '18 at 15:06
  • $\begingroup$ I was confused because your answer matches my understanding of what the Embedding layer does apart from calling it an "autoencoder". $\endgroup$ – kbrose Apr 3 '18 at 15:08
0
$\begingroup$

Embedding layer uses embedding matrix for mapping data and is never updated during training. There is no trainable parameters in the Keras Embedding layer. You can refer the Keras embedding layer docs for detailed understanding.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.