These last month I have been studying all about word embeddings and the most known pre-trained word embeddings, Word2Vec, GloVe, FastText, etc. I have read many times how important It is to take advantage of pre-trained models when doing a given task however I don't understand how a pre-trained model can adapt to my given corpus. Furthermore, If I have new words not present in the pre-trained model will I be able to use this pre-trained model to learn the embeddings for the new words?
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
I don't understand how a pre-trained model can adapt to my given corpus
You are correct in thinking this way. It is not a magic wand. It learns the embedding values based on the underlying context of the corpus(e.g. news) which may work in the broad sense but not in a specific case.
Two cities may get the embeddings based on their geographical location but that might not be the embedding we may like if we are comparing cities on crime rate/GDP etc.
One of the best posts on Word Embedding [Blog post by Sebastian Ruder] has mentioned all the limitations in detail.
An excerpt from the post,
"...One of the major downsides of using pre-trained embeddings is that the news data used for training them is often very different from the data on which we would like to use them. In most cases, however, we do not have access to millions of unlabelled documents in our target domain that would allow for pre-training good embeddings from scratch...."
If I have new words not present in the pre-trained model will I be able to use this pre-trained model to learn the embeddings for the new words?
It will get a value from a bucket for all the out-of-vocabulary(OOV) list. Sometimes it can be the same value for all the new words. But there is some known strategy to deal with the scenario. Check the post for relevant papers.
From the same posts,
"....One of the main problems of using pre-trained word embeddings is that they are unable to deal with out-of-vocabulary (OOV) words, i.e. words that have not been seen during training. Typically, such words are set to the UNK token and are assigned the same vector, which is an ineffective choice if the number of OOV words is large...."
But this issue will prevail even if you learn embeddings with your own corpus. So I believe it is not just limited to the pre-trained embeddings.
$\begingroup$
$\endgroup$
The brain of a model resides in its weights. Before any training happens - an empty model's weights are randomly initialized. The model training process then adjusts the weights into a more "favorable" region in N dimensional space.
So when you use pre-trained models - your model weights actually start from a "favorable" region (representing past learnings) instead of random selection (i.e. trying to begin from scratch). So that's why learning happens faster for pre-trained models - and you are likely to get good results if your training data looks similar to that which was used to train the original model.
$\begingroup$
$\endgroup$
I should add, since you mention FastText, that FastText uses subword information to build its word vectors. Subword information is not tied to any specific word and can therefore be used to create vectors for OOV or rare words (the authors of the FastText algorithm specifically mention the ability to cater to rare word vectors not encountered).
BERT, GPT,etc are the latest set of pre-trained models that have refined the vectors for their words by basically studying the words across very large and very numerous contexts e.g sentences, documents, etc in which the word could possibly be found and applying specialized attention mechanisms (transformers) to expand the context of the word as much as possible to refine the embedding of the word (BERT looks at upto 512 tokens at once in the context to determine the vector representation). BERT also uses subword information / a sentencepiece byte pair encoding algorithm to cater to OOV words.
BERT is especially meant to be used in a way that you can lift the embedding from pre-trained models for the words in the corpus you are studying and use those for your NLP tasks. These pre-trained embeddings are very powerful and have typically taken days, if not weeks, to fine tune across very large contexts. If you take a look at , for example a toolkit like HuggingFace (https://huggingface.co/), they have a lot of models with embeddings catered for specific tasks e.g one set for classification, one set for sequence labeling, etc.
