I am wondering which data cleaning steps should be performed if you want to re-fine a BERT model on custom text data.

Which steps should be performed?

Does it make sense to perform a stemming or lemmatization if it has not been applied to the initial training of the BERT Base/Large model?

  • $\begingroup$ Yes it answered 50% of my question. Thanks. I would like to know if the same pre-processing functions which have been applied on the training data of the BERT base model also need to be applied to the data which is used for the fine-tuning of the BERT model? Asking the other way round: If no preprocessing was performed on the training data of the base model... would it be a good idea to apply a preprocessing (like for example stemming) on the data for the (fine-tuning) training of the BERT model? $\endgroup$ Nov 26, 2020 at 23:20
  • $\begingroup$ No, it would not be a good idea to apply stemming on the fine-tuning data. For transfer learning to be effective, the fine-tuning data should resemble the original data used for pre-training. $\endgroup$
    – noe
    Nov 26, 2020 at 23:34
  • $\begingroup$ Thanks. This is what I wanted to know. $\endgroup$ Nov 27, 2020 at 7:59


Browse other questions tagged or ask your own question.