# Why using a frozen embedding layer in an LSTM model

I'm studying this LSTM mode: https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis

They use a frozen embedding layer which uses an predefined matrix with for each word a 300 dim vector which represents the meaning of the words.

As you can see here:

embedding_layer = Embedding(vocab_size, W2V_SIZE, weights=[embedding_matrix], input_length=SEQUENCE_LENGTH, trainable=False)


The embedding layer is frozen which means that the weights are not changed during training.

Why is this done?

• It depends whether you are using external word vectors or not, but a smarter way is to keep it fixed for few stating epochs and then train it as well for the last few epochs keeping the rest arch fixed in case you have OOM or whole arch is trained – Aditya Jun 3 '19 at 3:28

The embedding matrix which used in the initialization of the Embedding layer is highly trained on a large corpus of text. The training and the data are so huge that the embedding has learnt a type of association between words.

A pretrained embedding like Word2Vec will produce vectors for words like school and homework which are similar to each other in the embedding space.

Many such associations are learnt after rigorous training mostly on high-end machines and precisely calculated parameters.

Why is the Embedding layer set to trainable=false?

As mentioned in the code, we have given a pretrained embedding matrix to the Embedding layer through the weights= argument. As the word suggests, its "pretrained" and requires no additional training.

We can enjoy the benefits of such an embedding by keeping it untrainable. Additional training in the context of our task, may result in unusual behaviour of the Embedding layer and also distort the learned associations.

In some cases, the Embedding layer is kept trainable.

Another benefit of using a static (not training) Embedding layer is that it reduces bandwidth to the model.

In this case, there is a 300x reduction in bandwidth used. Instead of sending, for example, 300 floats corresponding with each value in the 300-dimensional vector, you can send a single integer.

If you're running your model on the same machine, it might not be a big deal, but if you're using TensorFlow Serving to host your model, the reduced bandwidth can be useful.