Twitter Sentiment Analysis: Detecting neutral tweets despite training on only Positive and Negative Classes

Question

I am a newbie when it comes to machine learning. I am trying to get hands on experience by analyzing different supervised learning algorithms using scikit-learn library of python. I am using the sentiment140 dataset of 1.6 million tweets for sentiment analysis using various of these algorithms.

I don't know if it is a stupid question, but I was wondering whether if it'd be possible to classify into three classes (positive, negative and neutral) when you've only trained over two classes (positive and negative). The sentiment140 training set consists of only two classes (positive and negative) of 1.6M tweets in total but their test set consists of 500 tweets over three classes (positive, negative and neutral), so it got me thinking.

So is this possible? If yes, how do I proceed to identify the neutral tweets? Intuitively, I can use to find the conditional probability of each classified tweet in the test set over each class (using predict_proba) and tell if it's neutral if it is below a certain threshold (say less than 0.7) for both the positive and negative classes. Is this the right way to go?

Answer 1

The quick (and not very satisfying) answer is "it depends" -- specifically it depends upon what your underlying conceptual model of human emotion is and how it manifests in verbal/written behavior.

What is your characterization of neutrality in relation to positive and negative valence? Can documents be put on some sort of quantitative scale with neutral sandwiched between positive and negative? This position has linguistic support at least with simple phrases which express single valence states (the solution is {great > good > acceptable > poor > horrible}).

As @dmb poster suggested, if this is your conceptual model, then you might reasonablly argue that neutral falls between pos/neg and all you have to do is determine the optimal boundaries/cutoff. Thus, you can assign neutral to a test case even if the classifier wasn't trained on neutral cases (though you do need some way of determining these cutoffs).

But what about more complex cases? What happens when you move from the level of the phrasal unit to larger sentential and discourse level units? How would you rate the sentence "I love apples but hate bananas"? Do the positive and negative elements cancel each other out to create a neutral? You can easily see how muddied this gets when we start to talk about real human texts. In my opinion, I don't think it is particularly meaningful to talk about a document-level emotion score. Rather, I believe people express emotion/sentiment that is directed at an individual objects "I hate bananas" -> hate(subj, obj) that get combined into discourse-level constructions.

So, yes, I think you could reasonable defend the use of outputting a neutral category if only pos/neg are used to train... BUT you will need to justify your use of particular cutoffs as well as consider how to deal with longer documents that express multiple (conflicting) emotions. My preferred course of action would be to have people read and label tweets as pos, neg, neutral, and mixed (then ensure inter-coder reliability) to create my training data... let the algorithm do the hard work of finding the cutoff values.

Answer 2

Say the results of one sample is $P(positive | sample) = 0.15$ and $P(negative | sample) = .05$, as opposed to just taking the higher probability group you can set a minimum threshold, say 0.3. Since both are below this you classify it as neutral

Answer 3

This really depends on how comprehensive a model you use. In most cases, the model for sentiment analysis is pretty simple and based on the "bag of words." In this case the positive and negative samples in the training set allow you, basically, to mine both positive and negative emotionally-loaded words.

Now imagine you have this job done perfectly but you have some text which contains neither positive nor negative words. This text can be considered neutral.

Another case is when the text contains positive and negative words in equal numbers. Potentially, this means the text contains both negative sentences and positive ones. Or there might be negations in the text that reverse the sentiment of some words. Could this text be considered neutral? Maybe. But simple "bag of words" models can't distinguish this case from the previous one.

All this means is that training data containing only positives and negatives could still recognize "neutral" texts. But you'll need to think carefully about neutrality criteria acceptable for your use case, and about the model you are going to use so that you get the exact behavior you want.

For more information on this topic, you might want to look at this blog post several colleagues and I wrote about creating training and test data sets: https://blog.griddynamics.com/creating-training-and-test-data-sets-and-preparing-the-data-for-twitter-stream-sentiment-analysis-of-social-movie-reviews