I'm finetuning a pretrained Huggingface model based on Transformers for a downstream Text Generation task, but I have doubts on how the fine-tuning process should be monitored:

In classification, I usually calculate the loss and other metrics (e.g accuracy) on a validation set to for early stopping and to save the checkpoint of the best epoch, but for Text Generation tasks I see these additional issues:

  • (Causal) text generation is slow, while training can be more parallelized with teacher forcing. Does it make sense to perform validation and generate text after every training epoch?

  • Text generation depends on an additional set of hyperparameters which greatly condition the quality of generated text, such as the decoding technique (greedy vs sampling-based or other techniques), number of beams, temperature, maximum length, etc. All these parameters are not actually used during training. I suppose also find the best combination after training, but how can I monitor the training?

  • HuggingFace generation API does not provide the loss during prediction, i.e I cannot generate text and calculate the cross-entropy loss (at least out-of-the-box) during validation. To calculate loss I could either

    1. Create a custom generation procedure which includes loss.
    2. Perform two passes on all data during validation (one with model.generate and one with model.forward )

    Both these alternatives are suboptimal and this made me think that it is not common to calculate validation loss in text generation tasks, is it true?

What is the common way to monitor training/fine-tuning of text generation models?

  • $\begingroup$ I think if your task is similar to the tasks those metrics were created for, you should use those standard metrics, and if it is not or they are not sufficient then it is more difficult and in order to answer your question more details would be needed. $\endgroup$
    – Valentas
    Commented Jun 9, 2023 at 8:12
  • $\begingroup$ @Valentas I think the question is not related to one single text generation task, rather if one can (or should) monitor both the training progress (with crossentropy) and the current quality of generated text (with n-gram based metrics or other types) $\endgroup$
    – Ciodar
    Commented Jun 9, 2023 at 13:57


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.