2
$\begingroup$

I'm studying this LSTM mode: https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis

They use a frozen embedding layer which uses an predefined matrix with for each word a 300 dim vector which represents the meaning of the words.

As you can see here:

embedding_layer = Embedding(vocab_size, W2V_SIZE, weights=[embedding_matrix], input_length=SEQUENCE_LENGTH, trainable=False)

The embedding layer is frozen which means that the weights are not changed during training.

Why is this done?

$\endgroup$
  • $\begingroup$ It depends whether you are using external word vectors or not, but a smarter way is to keep it fixed for few stating epochs and then train it as well for the last few epochs keeping the rest arch fixed in case you have OOM or whole arch is trained $\endgroup$ – Aditya Jun 3 at 3:28
4
$\begingroup$

The embedding matrix which used in the initialization of the Embedding layer is highly trained on a large corpus of text. The training and the data are so huge that the embedding has learnt a type of association between words.

Word Embeddings

A pretrained embedding like Word2Vec will produce vectors for words like school and homework which are similar to each other in the embedding space.

Many such associations are learnt after rigorous training mostly on high-end machines and precisely calculated parameters.

Why is the Embedding layer set to trainable=false?

As mentioned in the code, we have given a pretrained embedding matrix to the Embedding layer through the weights= argument. As the word suggests, its "pretrained" and requires no additional training.

We can enjoy the benefits of such an embedding by keeping it untrainable. Additional training in the context of our task, may result in unusual behaviour of the Embedding layer and also distort the learned associations.

In some cases, the Embedding layer is kept trainable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.