I've read many articles and papers mentioning how unsupervised training is conducted while pre-training a BERT model. I would like to know if it is possible to fine-tune a BERT model in an unsupervised manner or does it always have to be supervised?
The distinction between supervised and unsupervised is a little bit tricky here. BERT pre-training is unsupervised with respect to the downstream tasks, but the pre-training itself is technically a supervised learning task. BERT is trained to predict words that have been masked in the input, so the target words are known at training time. The term unsupervised fine-tuning is thus a little confusing here.
If we use BERT in a clever way (e.g., using a method called iPET), we can use its language modeling abilities to perform some tasks in an (almost) zero-shot setup, which basically means that BERT learned to perform the task in an unsupervised way. However, it is disputable, if this could be called unsupervised fine-tuning.
BERT can be of course fine-tuned in an unsupervised way by continued pre-training. It can be viewed as a way of domain adaptation of the model, which is typically again followed by supervised fine-tuning. Imagine you want to do a task on legal text, so you can first adapt BERT for legal text using large amounts of plain text, and fine-tune it using on much smaller labeled data.