I'm a bit confused the proper usage of Embedding layer in Keras for seq2seq purpose (I'd like to reconstruct the TensorFlow se2seq machine translation tutorial in Keras). My questions are the following:
I understand that Embedding layers turn word values in a sentence into fixed-dimension-long representation. But I observe two distinct usage of Embedding layers: one on one hand (like this tutorial on Keras Blog) utilizes external pre-trained word2vec vectors via the weights
parameter:
from keras.layers import Embedding
embedding_layer = Embedding(len(word_index) + 1,
EMBEDDING_DIM,
weights=[embedding_matrix],
input_length=MAX_SEQUENCE_LENGTH,
trainable=False)`
while in other cases there is no such an external output but users just leave to the Embedding layer to decide the representation vectors. I don't understand what is the real difference between these approaches regarding the desired outcome? Maybe the internal-only solution is not a semantic representation? What is the point of applying embedding layer to an external matrix of which the rows already have fix length?
Moreover, what is the purpose/effect of the trainable
parameter of the Embedding layer? Am I correct guessing that this set to True let the Embedding layer fine-tune the imported word2vec weights to take the actual training examples into consideration?
Further, how to instruct Embedding layer to properly encode "metacharacters"? Setting the mask_zero
parameter True it can incorporate padding zeroes but what about UNK (unknown), EOS (End of Sentence)? (By the way, I cannot understand what is the point to explicitly sign the end of sentence in a sentence based input...)
And finally: how could a model predict the translation of a word which is not represented in the training set? Is it tries to approximate it with the "closest" one in the vocabulary?