When to use GloVe vocabulary vs. building a vocabulary from the training data?

Question

While studying some (pytorch) examples that use pretrained GloVe vectors I came across two variants:

1. Use the vocabulary of the GloVe vectors and thus initialize the embedding layer with the pretrained GloVe vectors.
2. Build a vocabulary from the corpus and then only use the pretrained GloVe vectors that correspond to that vocabulary to initialize the embedding layer.

To me it seems that by using the vocabulary of the GloVe vectors there is a chance that some of the tokens in the training set may not have corresponding GloVe vectors and are thus excluded from the vocabulary. Therefore, you may miss out on tokens that are significant for the task.

On the other hand, building a vocabulary from the corpus implies that the model cannot handle unseen words (as far as I understand that correctly).

Therefore, I was wondering: When should one use the GloVe vocabulary vs. building a vocabulary from the training data? And would it make sense to use the "union" of the two vocabularies instead?

Answer 1

The arguments that you say are pretty much correct.

The primary reason for using pre-trained embeddings is typically the lack of task-specific training data. In tasks that (at least for some languages) have vast amounts of training data, such as machine translation, the word embeddings are always trained jointly with the rest of the model.

The mere fact that you are using GloVe indicates that your training data is not that large. In that case, it is very likely that at inference time, words can appear that were not in the training data, but GloVe still has a good representation for them and the rest of the model knows how to use it. GloVe has a really large vocabulary, it is much more likely that there will be GloVe words at inference time than unknown training words.

You can also opt for a hybrid solution. If there are words in the training data, that are not in GloVe, but still are frequent enough, you can learn their embeddings jointly with the rest of the model while keeping the rest of the GloVe embeddings frozen. (But implementing this might be a little bit tedious.)