0
$\begingroup$

I am reading a paper, where the authors assess online public sentiment in China in response tot the government's policies during Covid-19, using a Chinese BERT model. The author's objective is not only to learn whether a given online post is critical or supportive, but also learning to whom each post was directed at (e.g. CCP, local governments, health ministry, etc). To achieve this, the authors further state in pages 8 through 9, that they,"To train the classifer, we randomly sample approximately 5,000 posts from each dataset (10,541 posts in total), stratified by post creation data. This sample is used for a number of analyses, and we refer to it as the Hand-Annotated Sample."

My question here is what's the value of using human-annotated posts in combination with a BERT sentiment analysis model?

Specifically, my understanding of BERT as a technique is that it eliminates or at least minimizes the need for pre-labelling a sample of text for sentiment analysis purposes, and it's not clear to me why we still need hand-annotated text by humans even when using BERT.

$\endgroup$

1 Answer 1

1
$\begingroup$

BERT is pre-trained on two generic tasks: masked language modeling and next sentence prediction. Therefore, those tasks are the only things it can do. If you want to use it for any other thing, it needs to be fine-tuned on the specific task you want it to do, and, therefore, you need training data, either coming from human annotations or from any other source you deem appropriate.

The point of fine-tuning BERT instead of training a model from scratch is that the final performance is probably going to be better with BERT. This is because the weights learned during the pre-training of BERT serve as a good starting point for the model to accomplish typical downstream NLP tasks like sentiment classification.

In the article that you referenced, the authors describe that they fine-tune a Chinese BERT model on their human-annotated data multiple times separately:

  1. To classify whether a Weibo post refers to COVID-19 or not.
  2. To classify whether posts contained criticism or support.
  3. To identify posts containing criticism directed at the government or not.
  4. To identify posts containing support directed at the government or not.

Fine-tuning BERT usually gives better results than just training a model from scratch because BERT was trained on a very large dataset. This makes the internal text representations computed by BERT more robust to infrequent text patterns that would be hardly present in a smaller training set. Also, dictionary-based sentiment analysis tends to give worse results than fine-tuning BERT because a dictionary-based approach would hardly grasp the nuances of language, where not only does a "not" change all the meaning of a sentence, but any grammatical construction can give subtle meaning changes.

$\endgroup$
3
  • 1
    $\begingroup$ Thanks for your thorough and clear explanation! I would appreciate it if you can elaborate further as to why using BERT, along with a human-annotated sample, is superior to say.. dictionary-based sentiment analysis? Or simply relying strictly on hand-annotation as in this paper: scholar.google.com/… $\endgroup$ Jul 20, 2022 at 15:55
  • $\begingroup$ I updated my answer to address the doubts you expressed in your comment. $\endgroup$
    – noe
    Jul 21, 2022 at 9:37
  • $\begingroup$ Thanks, great explanation! $\endgroup$ Jul 21, 2022 at 10:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.