0
$\begingroup$

I have gained a basic understanding of using BERT for various NLP/text mining tasks. When it comes to fine-tuning of BERT, I always see that fine-tuning is performed using some classification tasks. So, how should I refine the word/sentence embeddings vector given by the BERT model in the case when I have a set completely unlabelled set of documents? I'm aware that the BERT model is originally trained on unlabelled data, so there must be some way.

$\endgroup$
2
$\begingroup$

So, how should I refine the word/sentence embeddings vector given by the BERT model in the case when I have a set completely unlabelled set of documents?

What are you looking to achieve with these unlabelled documents? If you are looking to classify them, then there is no way of getting around getting labels and fine-tuning on them.

I'm aware that the BERT model is originally trained on unlabelled data, so there must be some way.

If you don't need to perform any specific end task (like classification) with the data, and instead is just looking to train BERT using your own data, then there is a way. BERT is self-supervised, meaning that it takes unlabelled data and automatically generates labels that it can train on. The same methods can be used to train on any large language corpus. However, this will only train the embeddings and fine-tuning on labels will still always be needed for classification.

I think huggingfaces' blog post is a good starting point if you want train a general BERT on your own dataset.

You can read about how BERT is trained on generated labels from text here.

$\endgroup$
1
  • $\begingroup$ Thanks Simon for your reply. I actually want to extract word vectors using BERT which are fine - tuned with respect to my dataset. Then i''ll further use them to perform document clustering on the dataset. $\endgroup$ May 27 '20 at 5:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.