Yes, it is possible to train an RNN-based architecture like GRU or LSTM with random sentences from a large corpus to learn word embeddings.
The word embeddings of the corpus words can be learned while training a neural network on some task e.g. sentiment classification. Before it can be presented to the RNN, each word is first encoded so that it is represented by a unique integer e.g. using a tokenizer. We add a padding token to make all the sentences of the same length.
Doing this is referred to as building an embedding layer "in front of" your LSTM/RNN/GRU network model. For the embedding layer you need to specify:
- maximum length of a sequence
- embedding size for each token.
The drawback is that using this technique, training will take more time because you have to not only train your prediction model but also your word embeddings. This is because the total amount of trainable parameters will be larger as compared to a model where we use a pre-trained word embedding with frozen parameters.
Finally, depending on your dataset size using pre-trained embeddings trained on some larger corpus will lead to better embeddings than training your own word embeddings on your smaller dataset.
Here is an experiment using both approaches: