Are word embeddings further updated during training for document classification?

Question

I am relatively new to the area of using word embeddings in NLP tasks. From a large corpus of documents, I train word2vec word embedding vectors and afterwards I am going to use these for document classification, combined with RNN based classifiers (LSTM, GRU), which is a pretty standart pipeline nowadays.

There is one issue; that should we update the word embeddings during the document classification training as well.

I am used to tasks like image classification/object detection in the past. You get an image input and the convolution features extracted from that image are updated during the numerical optimization of a CNN. But the image itself is never updated, naturally, since it is the original data.

How do we treat the embedding vectors in the world of text documents? They are not "natural" exactly like the images, we learn them from an unsupervised method first (word2vec, GloVe or any other tool) so I think they can be further fine-tuned during the supervised training. Is it the common practice to update the embedding vectors as well as the RNN parameters during the training of the sequence classifier or should we leave them as constant (in order to avoid overfitting) ?

Answer 1

Word embeddings are generally used as input features, which like you noticed for image based models, do not get modified during training.

It is in fact quite difficult to update embeddings during training (or at all after they have been computed!), because they are often in some latent space that holds information regarding their relationships to one another. It is for this reason that adding new vocabulary to existing embeddings is extremely challenging, and updating computed embeddings is also difficult without using all the data that was originally used to create them.

I am afraid I don't have an example I can point to where embedding are used during training and simultaneously updated/improved upon. I can imagine that there are ways to store them, however, such that it is technically feasible (although perhaps memory intensive!)