I've read many articles and papers mentioning how unsupervised training is conducted while pre-training a BERT model. I would like to know if it is possible to fine-tune a BERT model in an unsupervised manner or does it always have to be supervised?


1 Answer 1


The distinction between supervised and unsupervised is a little bit tricky here. BERT pre-training is unsupervised with respect to the downstream tasks, but the pre-training itself is technically a supervised learning task. BERT is trained to predict words that have been masked in the input, so the target words are known at training time. The term unsupervised fine-tuning is thus a little confusing here.

If we use BERT in a clever way (e.g., using a method called iPET), we can use its language modeling abilities to perform some tasks in an (almost) zero-shot setup, which basically means that BERT learned to perform the task in an unsupervised way. However, it is disputable, if this could be called unsupervised fine-tuning.

BERT can be of course fine-tuned in an unsupervised way by continued pre-training. It can be viewed as a way of domain adaptation of the model, which is typically again followed by supervised fine-tuning. Imagine you want to do a task on legal text, so you can first adapt BERT for legal text using large amounts of plain text, and fine-tune it using on much smaller labeled data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.