I'm studying this LSTM mode: https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis

They use a frozen embedding layer which uses an predefined matrix with for each word a 300 dim vector which represents the meaning of the words.

As you can see here:

embedding_layer = Embedding(vocab_size, W2V_SIZE, weights=[embedding_matrix], input_length=SEQUENCE_LENGTH, trainable=False)

The embedding layer is frozen which means that the weights are not changed during training.

Why is this done?

  • $\begingroup$ It depends whether you are using external word vectors or not, but a smarter way is to keep it fixed for few stating epochs and then train it as well for the last few epochs keeping the rest arch fixed in case you have OOM or whole arch is trained $\endgroup$
    – Aditya
    Jun 3 '19 at 3:28

The embedding matrix which used in the initialization of the Embedding layer is highly trained on a large corpus of text. The training and the data are so huge that the embedding has learnt a type of association between words.

Word Embeddings

A pretrained embedding like Word2Vec will produce vectors for words like school and homework which are similar to each other in the embedding space.

Many such associations are learnt after rigorous training mostly on high-end machines and precisely calculated parameters.

Why is the Embedding layer set to trainable=false?

As mentioned in the code, we have given a pretrained embedding matrix to the Embedding layer through the weights= argument. As the word suggests, its "pretrained" and requires no additional training.

We can enjoy the benefits of such an embedding by keeping it untrainable. Additional training in the context of our task, may result in unusual behaviour of the Embedding layer and also distort the learned associations.

In some cases, the Embedding layer is kept trainable.


This slide would be helpful for you.

enter image description here

In simple words, the pretrained model are trained over a large dataset and all words in the vocabulary have already been trained perfectly, but when you finetune the model your dataset is relatively small and many words would not be retrained then the relations between the retrained and unretouched words would be a mess.

In the slide the word television is not retrained and it stays where it was but the retrained TV swims away then the distance between TV and television increases. This can be bad.


Another benefit of using a static (not training) Embedding layer is that it reduces bandwidth to the model.

In this case, there is a 300x reduction in bandwidth used. Instead of sending, for example, 300 floats corresponding with each value in the 300-dimensional vector, you can send a single integer.

If you're running your model on the same machine, it might not be a big deal, but if you're using TensorFlow Serving to host your model, the reduced bandwidth can be useful.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.