Summing three lexicon based approach methods for sentiment analysis?

Question

I'm doing sentiment analysis using a lexicon based approach and I have a bunch of news headlines that needs to be categorized as negative, positive and neutral or within a scale ranging from -1 (very negative sentiment) to +1 (very positive sentiment).

I'm considering using three different methods: TextBlob, Vader (Valence Aware Dictionary and Sentiment Reasoner) and LIWC2015 as I have access to it. This would lead to three predicted sentiment of each headline.

I hove two question:

• Which are the pros and cons of this technique?
• And, once runned the 3 analisys, how can I melt all them into a single metrics? Is standardize/normalize the 3 metrics and averaging them a good solution?

Answer 1

What you're proposing is some very simple form of ensemble learning. You need to have at least a sample of labelled data in order to evaluate any method. Using this labelled data you can:

• evaluate each of the three methods on their own
• evaluate your idea of averaging the 3 methods predictions
• if you have enough labelled data, you could even train a model which combines their predictions optimally (this would be full-fledged stacked generalization)

Answer 2

Which are the pros and cons of this technique?

Context insensitivity: the libraries you mention are intended for general sentiment analysis so you could encounter some false positive/false negative issues.

• False positives: words with a particular sentiment in the dictionary that doesn't apply to headlines –e.g. "low" may have a negative connotation in general but could be even positive in "covid cases remain low").
• False negatives: words that don't usually carry sentiment information but do in headlines, e.g. "New Brexit deal on the table" would be negative in a context in which we assume there was already a deal and should remain effective.

Low coverage: if I remember correctly, the tools you mention are all dictionary-based or pre-trained. It is unlikely that those dictionaries/training have been optimized for your use case. Usually, you're better off training your own classifier on domain-specific data.

Underfitting: as a far as I know, those libraries won't be able to handle phenomena like logical negation or polarity-reversal systematically. In some cases, they may include n-gram information that allows them to discriminate "i_like" as positive and "i_don't_like" as negative, but will probably be unable to handle most long-tail cases like "signed_the_deal" and "didn't_sign_the_deal" in "uk_signed_the_brexit_deal". Polarity reversal refers to cases like "low" in the opposition between "low number of covid cases" (typically good –unless your domain is medical research on Covid and more cases are better because they represent more data) and "low confidence in the Spanish government" (typically bad unless you're a hedge fund manager betting against the Spanish economy).

And, once runned the 3 analisys, how can I melt all them into a single metrics? Is standardize/normalize the 3 metrics and averaging them a good solution?

I totally agree with Erwan's comment, you should collect a sample of test cases that you can use for supervised evaluation and parameter fine-tuning.

As potential risks of the approach you suggest, make sure the scores you get from the different models are neither completely correlated (in that case, you can just use the best/fastest model) or orthogonal (in the unlikely case some models often contradict each other, you should find out which ones are "lying").

There is another potential risk, more interesting: the case in which all models are complementary of each other and cover different sub-spaces of the sentiment domain: if one model is particularly good at picking up sentiment in politics headlines, another in financial headlines, and another in sports headlines, you could end up with a significant number of predictions in which the correct (non-neutral) prediction has to compete with two incorrectly neutral predictions and would always be averaged out, potentially resulting in lots of false negatives. An alternative approach to averaging would be to select only the highest-confidence prediction, particularly if the other two are neutral.