Limitations of NLP BERT model for sentiment analysis

Question

I am reading a paper, where the authors assess online public sentiment in China in response tot the government's policies during Covid-19, using a Chinese BERT model. The author's objective is not only to learn whether a given online post is critical or supportive, but also learning to whom each post was directed at (e.g. CCP, local governments, health ministry, etc). To achieve this, the authors further state in pages 8 through 9, that they,"To train the classifer, we randomly sample approximately 5,000 posts from each dataset (10,541 posts in total), stratified by post creation data. This sample is used for a number of analyses, and we refer to it as the Hand-Annotated Sample."

My question here is what's the value of using human-annotated posts in combination with a BERT sentiment analysis model?

Specifically, my understanding of BERT as a technique is that it eliminates or at least minimizes the need for pre-labelling a sample of text for sentiment analysis purposes, and it's not clear to me why we still need hand-annotated text by humans even when using BERT.

Answer 1

BERT is pre-trained on two generic tasks: masked language modeling and next sentence prediction. Therefore, those tasks are the only things it can do. If you want to use it for any other thing, it needs to be fine-tuned on the specific task you want it to do, and, therefore, you need training data, either coming from human annotations or from any other source you deem appropriate.

The point of fine-tuning BERT instead of training a model from scratch is that the final performance is probably going to be better with BERT. This is because the weights learned during the pre-training of BERT serve as a good starting point for the model to accomplish typical downstream NLP tasks like sentiment classification.

In the article that you referenced, the authors describe that they fine-tune a Chinese BERT model on their human-annotated data multiple times separately:

1. To classify whether a Weibo post refers to COVID-19 or not.
2. To classify whether posts contained criticism or support.
3. To identify posts containing criticism directed at the government or not.
4. To identify posts containing support directed at the government or not.

Fine-tuning BERT usually gives better results than just training a model from scratch because BERT was trained on a very large dataset. This makes the internal text representations computed by BERT more robust to infrequent text patterns that would be hardly present in a smaller training set. Also, dictionary-based sentiment analysis tends to give worse results than fine-tuning BERT because a dictionary-based approach would hardly grasp the nuances of language, where not only does a "not" change all the meaning of a sentence, but any grammatical construction can give subtle meaning changes.