How to user Keras's Embedding Layer properly?

Question

I'm a bit confused the proper usage of Embedding layer in Keras for seq2seq purpose (I'd like to reconstruct the TensorFlow se2seq machine translation tutorial in Keras). My questions are the following:

I understand that Embedding layers turn word values in a sentence into fixed-dimension-long representation. But I observe two distinct usage of Embedding layers: one on one hand (like this tutorial on Keras Blog) utilizes external pre-trained word2vec vectors via the weights parameter:

from keras.layers import Embedding

embedding_layer = Embedding(len(word_index) + 1,
  EMBEDDING_DIM,
  weights=[embedding_matrix],
  input_length=MAX_SEQUENCE_LENGTH,
  trainable=False)`

while in other cases there is no such an external output but users just leave to the Embedding layer to decide the representation vectors. I don't understand what is the real difference between these approaches regarding the desired outcome? Maybe the internal-only solution is not a semantic representation? What is the point of applying embedding layer to an external matrix of which the rows already have fix length?

Moreover, what is the purpose/effect of the trainable parameter of the Embedding layer? Am I correct guessing that this set to True let the Embedding layer fine-tune the imported word2vec weights to take the actual training examples into consideration?

Further, how to instruct Embedding layer to properly encode "metacharacters"? Setting the mask_zero parameter True it can incorporate padding zeroes but what about UNK (unknown), EOS (End of Sentence)? (By the way, I cannot understand what is the point to explicitly sign the end of sentence in a sentence based input...)

And finally: how could a model predict the translation of a word which is not represented in the training set? Is it tries to approximate it with the "closest" one in the vocabulary?

Answer 1

You've brought up some very good points. Let's walk through all of this:

A word embedding is a mathematical representation of a word. This is needed since we cant work with text as plain input.

In order to get these word embeddings, there a different ways, methods and settings on how to calculate them. cbow, skip-gram and so on.

There are different pretrained word embeddings out there e.g.

• Fasttext 2017, trained on Wikipedia; cudos to Bojanowski, Piotr and Grave, Edouard and Joulin, Armand and Mikolov, Tomas
• Word2Vec 2013, trained Google News dataset; cudos to Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean
• Glove 2013, Wikipedia 2014 + Gigaword 5; cudos to Jeffrey Pennington, Richard Socher, Christopher D. Manning

This is just an excerpt of the most well-known ones. As you can see, they used different data sets - thus different word vocabulary and word embeddings respectively.

I don't understand what is the real difference between these approaches regarding the desired outcome? Maybe the internal-only solution is not a semantic representation? What is the point of applying embedding layer to an external matrix of which the rows already have fix length?

Keras is an awesome toolbox and the embedding layer is a very good possibility to get things up and running pretty fast. Convert the text into one-hot/count matrix, use it as the input into the word embedding layer and you are set.

On the other hand if you use pre-trained word vectors then you convert each word into a vector and use that as the input for your neural network. This approach would give you more flexibility when it comes to feature engineering.

As mentioned above, pre-trained word vectors were given mostly general text data sets. You might run into the point where you have some special kind of data (e.g. Tweets) where people write or behave differently. So you might look into training your own embeddings, on your own dataset - at the end of the day it depends on your task/problem and the metrics that you are tuning towards.

Moreover, what is the purpose/effect of the trainable parameter of the Embedding layer?

As you said correctly, it is to retrain the weights of the embeddings with the data set you use.

I cannot understand what is the point to explicitly sign the end of sentence in a sentence based input

One of the most important things in NLP is feature engineering. It is the same as you sitting in school and learning a language, what needs to be considered, vocabulary, it's grammar and rules. Things that makes it easier for you as a human being to understand the language. The same is applied here. You can see it as one part of feature engineering, it all sums up to the bigger picture.

And finally: how could a model predict the translation of a word which is not represented in the training set?

Either you convert the word that couldn't be found to an <UNK> token (unknown word) which then represents its group. However it requires having the <UNK> token in the trained word embeddings. Or you could use fasttext binary file, which calculates word vectors on the fly for unknown words.