In many of the sources I have found regarding text generation with word-based RNN models (LSTM or GRU), the model is trained to perform a classification task across the vocabulary (such as with categorical cross-entropy loss) to predict the next word. An example can be found here for starters. Over a large vocabulary, this gets computationally expensive.
To me, it seems much more practical to first get contextual embeddings for each word in the training/testing dataset by using a pre-trained model like BERT. Then the sequential model could predict words using a loss function that measures the distance between predicted and actual embeddings with MSE or cosine similarity (CosineEmbeddingLoss). A lookup in the embedding space could return the word nearest to each prediction to make the output human-readable.
Is there anything wrong with the outlined approach or is it viable for text generation? The softmax operator and the classification task seem needlessly expensive for large vocabularies. Although BERT cannot be used to directly generate text (can bert be used for sentence generating tasks) I see nothing wrong with training a new model using BERT's embeddings or the embeddings of a similar model (see "BERT for feature extraction" here).