1
$\begingroup$

In many of the sources I have found regarding text generation with word-based RNN models (LSTM or GRU), the model is trained to perform a classification task across the vocabulary (such as with categorical cross-entropy loss) to predict the next word. An example can be found here for starters. Over a large vocabulary, this gets computationally expensive.

To me, it seems much more practical to first get contextual embeddings for each word in the training/testing dataset by using a pre-trained model like BERT. Then the sequential model could predict words using a loss function that measures the distance between predicted and actual embeddings with MSE or cosine similarity (CosineEmbeddingLoss). A lookup in the embedding space could return the word nearest to each prediction to make the output human-readable.

Is there anything wrong with the outlined approach or is it viable for text generation? The softmax operator and the classification task seem needlessly expensive for large vocabularies. Although BERT cannot be used to directly generate text (can bert be used for sentence generating tasks) I see nothing wrong with training a new model using BERT's embeddings or the embeddings of a similar model (see "BERT for feature extraction" here).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The main difference between RNN-based text generation and BERT is the attention mechanism based on transformers.

This attention mechanism is very important to add context between words and explains why the results are better than RNN in many applications.

However, in terms of text generation, GPT-2 is more adapted than BERT because it uses a masked self-attention mechanism. The mask self-attention model is trained by guessing the next word or token, contrary to BERT which is trained on whole phrases. That allows the definition of the next sequence of words in terms of probability, thanks to the previous ones.

Here is an article that explains how GTP2 works: https://jalammar.github.io/illustrated-gpt2/

The model is available here: https://huggingface.co/gpt2

Improve this answer
$\endgroup$
3
  • $\begingroup$ While transformer models rely on their attention mechanisms in lieu of recurrence, I don't see how it is an issue for my outlined approach. Prior to training my RNN-like model, I will use BERT to get contextual embeddings for each word in my dataset, including the words that come before each word (never after) to form embeddings. Attention won't look forward, and therefore shouldn't be an issue. The recurrent model will predict next-word embeddings from the input of prior word embeddings. Is this method viable or am I missing something? $\endgroup$
    – twiddler
    Jul 11, 2022 at 18:18
  • $\begingroup$ Your method seems viable, but I thought you would find GPT-2 better to fit your needs because you have the next word embedding and also an attention mechanism. $\endgroup$
    – Nicolas Martin
    Jul 11, 2022 at 18:44
  • 1
    $\begingroup$ I already had a prior text generation model trained using RNN. I was mainly looking to improve it rather than fine-tune a bigger model, but yes, GPT-2 may better suit my needs. I'll look into that :) $\endgroup$
    – twiddler
    Jul 11, 2022 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.