The common way of learning word embeddings is based on BOW, and Skip-gram models.
Is it possible to train a RNN-based architecture like GRU or LSTM with random sentences from a large corpus to learn word embeddings? Basically, we train a network with positive and negative samples, and back-propagate into word vectors. What are the drawbacks of this technique?
Any reference to similar works is highly appreciated.
Yes, it is possible to train an RNN-based architecture like GRU or LSTM with random sentences from a large corpus to learn word embeddings.
The word embeddings of the corpus words can be learned while training a neural network on some task e.g. sentiment classification. Before it can be presented to the RNN, each word is first encoded so that it is represented by a unique integer e.g. using a tokenizer. We add a padding token to make all the sentences of the same length.
Doing this is referred to as building an embedding layer "in front of" your LSTM/RNN/GRU network model. For the embedding layer you need to specify:
The drawback is that using this technique, training will take more time because you have to not only train your prediction model but also your word embeddings. This is because the total amount of trainable parameters will be larger as compared to a model where we use a pre-trained word embedding with frozen parameters.
Finally, depending on your dataset size using pre-trained embeddings trained on some larger corpus will lead to better embeddings than training your own word embeddings on your smaller dataset.
Here is an experiment using both approaches: https://towardsdatascience.com/machine-learning-word-embedding-sentiment-classification-using-keras-b83c28087456
