I'm studying this LSTM mode: https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis
They use a frozen embedding layer which uses an predefined matrix with for each word a 300 dim vector which represents the meaning of the words.
As you can see here:
embedding_layer = Embedding(vocab_size, W2V_SIZE, weights=[embedding_matrix], input_length=SEQUENCE_LENGTH, trainable=False)
The embedding layer is frozen which means that the weights are not changed during training.
Why is this done?
The embedding matrix which used in the initialization of the Embedding layer is highly trained on a large corpus of text. The training and the data are so huge that the embedding has learnt a type of association between words.
A pretrained embedding like Word2Vec will produce vectors for words like
schoolandhomeworkwhich are similar to each other in the embedding space.
Many such associations are learnt after rigorous training mostly on high-end machines and precisely calculated parameters.
Why is the Embedding layer set to trainable=false?
As mentioned in the code, we have given a pretrained embedding matrix to the Embedding layer through the weights= argument. As the word suggests, its "pretrained" and requires no additional training.
We can enjoy the benefits of such an embedding by keeping it untrainable. Additional training in the context of our task, may result in unusual behaviour of the Embedding layer and also distort the learned associations.
In some cases, the Embedding layer is kept trainable.
This slide would be helpful for you.
In simple words, the pretrained model are trained over a large dataset and all words in the vocabulary have already been trained perfectly, but when you finetune the model your dataset is relatively small and many words would not be retrained then the relations between the retrained and unretouched words would be a mess.
In the slide the word television is not retrained and it stays where it was but the retrained TV swims away then the distance between TV and television increases. This can be bad.
Another benefit of using a static (not training) Embedding layer is that it reduces bandwidth to the model.
In this case, there is a 300x reduction in bandwidth used. Instead of sending, for example, 300 floats corresponding with each value in the 300-dimensional vector, you can send a single integer.
If you're running your model on the same machine, it might not be a big deal, but if you're using TensorFlow Serving to host your model, the reduced bandwidth can be useful.
