I've pretrained the RoBERTa model with new data using a 'simpletransformers' library:

from simpletransformers.classification import ClassificationModel

OUTPUT_DIR = 'roberta_output/'
model = ClassificationModel('roberta', 'roberta-base',use_cuda=False, num_labels=22,
                        args={'overwrite_output_dir':True, 'output_dir':OUTPUT_DIR})


result, model_outputs, wrong_predictions = model.eval_model(test_df) # model evaluation on test data

where 'train_df' is a pandas dataframe that consists of many samples (=rows) with two columns: the 1st column is a text data - input; the 2nd column is a category (=label) - output.

I need to create the same model and pretrain it as above but using 'PyTorch' library instead of 'Simpletransformers' library. Is there any way to make it simple as the code above?

I've loaded the pretrained model as it was said here:

import torch
roberta = torch.hub.load('pytorch/fairseq', 'roberta.large', pretrained=True)
roberta.eval()  # disable dropout (or leave in train mode to finetune)

I also changed the number of labels to predict in the last layer:

roberta.register_classification_head('new_task', num_classes=22)

But, I can't find how I can pretrain the classifier with my 'train_df'. The only way I've found so far is from here where we use a PyTorch toolkit 'fairseq' and fairseq cli to pretrain RoBERTa model. Is this the only option or can it be done more simply?


2 Answers 2


I've recently been struggling with this type of problem. I followed and modified this tutorial from HuggingFace (which is deprecated by now because it uses an older version of transformers). The final result is here. Hope you find it helpful.

  • $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review $\endgroup$
    – Ethan
    Nov 4, 2022 at 0:30
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Nov 10, 2022 at 12:17

First of all, you are not pre-training the model. You are fine-tuning on your data. Second of all, you can follow the https://huggingface.co/docs/transformers/training It has great way of showing how you can use either pytorch or tensorflow framework to fine-tune your model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.