Sentiment Analysis models trained on articles / alternative data

Question

For a 6 class sentence classification task (emotion), I have a list of sentences where I retrieve the sentiment using a language model that was trained on Tweets (bertweet).

It works fine for simplistic sentences where the sentiment is also obvious (someone died, someone won something, someone was afraid of something, etc). However, when applying it to articles, it shows uncontrollable behavior.

Two examples of the sadness class:

How Your Family Can Volunteer During the Pandemic: 99% probability of sadness
There was a massacre in Bosnia where many were slaughtered: 96% probability of sadness

I have tried removing the softmax to break down the probabilities into absolute values in order to see if there's a difference there, but it seems that it is marginal and the first sentence again is "sadder" than the second one about the massacre.

There are many more such examples for all the other classes. Is there any model that is trained on articles? Possibly click bait titles and the kinds?

Answer 1

So, I think the interpretation of the model is part of the problem.

For the first: "How Your Family Can Volunteer During the Pandemic" them model is not 99% sad. It is 99% confident based on your training, that the context of this sentence is sad.

The same holds true for the second sentence: "There was a massacre in Bosnia where many were slaughtered" the model is 96% confident that sad is the sentiment of this observation.

Those probabilities are not intensities of the sentiment. So, if you are finding that globally your model is performing poorly on many or most test cases, it suggests one of a few things:

1. your model is not robust enough or trained on text of different length with different contextually complexities or there is too little training data

2. your labels are not appropriate to the sentiment you expect. Remember what is considered sad is determined by your input labels.

3. that massacre and slaughter might not be in the training set, or fully represented or used together such that when combined you get a high probability of sadness

In my experience this happens often when people cherry pick sentences and say, "look how awful this NLP model was on this: X." It happens frequently at my job. But remember, a neural network is designed to emulate a human neural network with appropriate training. Human beings are constantly taking in input and augmenting our training. Subtlety and consistency of our assessment of sentiment is the result of billions of exposures. Your model only has what you have trained it on.

Expecting it to always be right is not reasonable. Also it is not the case that the probability of a sentiment will scale relative to our sense of intensity. The goal of the model is to pick a class. That is done by identifying a sentiment and giving you a probability.

As the creator of the model, you decide where the cutoff value is.

As for the model itself, you are showing what look like titles.

Are you training on article titles or the article itself?

Titles and articles are not the same thing linguistically. Using a model that was designed for tweets to assess long-form copy also might not provide the best generalization capabilities depending on your methods.

In my personal experiences, the labeling itself and the pre-work (wrangling, grooming and such) which you do on language models (including the relative similarity in size and shape of training copy) has a lot more to do with results than small changes in the model structure.

I would start by looking at the global accuracy and then look at all the cases where if fails and see if there is something missing in the training data that you can augment to correct the erroneous predictions.