4
$\begingroup$

I wanted to perform Twitter sentiment analysis on Twitter tweets. When I googled it, I found a stanfordNLP code of sentiment class. I used it for finding Twitter sentiment later I came to know TwitterGate model is made for Twitter tweets only.

Then I replaced the stanfordNLP model with TwitterGate model. I did not find any difference in the sentiment results. Both models give the same result.

However, when I tested the same tweets with Monkeylearn API and Datumbox API sentiment, I got confused with results. In NLP and twitterGate most tweets sentiment is given as negative, where the same tweets sentiment is given as neutral in Monkeylearn & Datumbox APIs code.

How can I know which model is giving the correct sentiment?

$\endgroup$
5
$\begingroup$

A couple of important points:

  • Sentiment analysis is not an exact science. Two people, reading the same text in different contexts will come to different conclusions about sentiment, especially on borderline cases. Perhaps text has complex grammar, or has a metaphor or simile in it where it helps to understand what is actually being compared.

  • The ground truth for sentiment data sets is established by people. The best you can hope for from any trained ML classifier is that it closely matches the opinion of those people when it predicts on new data.

  • Your different sentiment analysis tools could well vary in quality, because they will have been built with different technologies at different times, and with different training data.

There are a couple of ways I can think of that you could assess the classifiers for your purpose:

1. Test against your own labelled test data

It is important to source this yourself to ensure it does not overlap with training data used by any of the models you want to compare (if it did overlap then it gives a big advantage to any model that trained using it). You will need to collect ground truth data on sentiment for all the texts.

Before you test, it is a good idea to pick a metric that you care about for your intended use. Accuracy is not the only metric, but it is probably fine for your purpose. You might choose other metrics if there are different costs associated with incorrect classifications - e.g. if assigning negative sentiment needs to be done cautiously.

Then run each classifier across your test set and calculate the metric. The classifier with the best metric is your best guess at the one you should use.

If you have a large enough test set, you can split it into parts and get a measure of the estimate error in your metric. This will help you see whether differences in the test metrics are significant.

This can be the best approach, especially if you can source data that you know matches your project goals. However, it would take a long time and a lot of effort.

If you are in a hurry, you could just search for and use someone else's dataset e.g I found these ones on a quick web search: a Kaggle competition and a free sample hosted by Niek Sanders (I have no idea of the quality of these - you should sample them and see if the data is useful for you). There is a risk though that these data sets were used to train one of your classifiers, so it would give you a false high rating.

2. Read up on associated reports and papers

Each of the sentiment analysers should explain the model class and data used to train it in documentation. Quite often you can find papers comparing the different approaches and quoting accuracy scores on a standard data set. If you are lucky, you will find enough comparisons that you get some sense of which algorithms are considered "cutting edge" and which are out-dated.

3. Collect text that is classified differently by algorithms you are considering, and get feedback

A variation of (1), you can do this at any stage, and it might work well in the context of a live product. For all the data that is classified, log how different classifiers respond to it, and where there is discrepancy, save it for later assessment. You will need some people to read and help classify the text (ideally without seeing what the classifiers thought), and you can collect metric data over time and use the classifier that does best once you have collected enough examples to make a clear decision.


Ultimately what matters is the results of using your classifier in a project. Just labelling data has no inherent purpose. It is the consequences of those labels that matter. So you need to be driven by results. If you use one classifier and get feedback that the product is not performing well, you may be able to trial another one. If your product is used by many people, and enough are giving feedback of quality, you could even A/B test a couple of classifiers in production.

$\endgroup$
4
$\begingroup$

You can classify a few of the tweets yourself, and compare afterwards which of the two algorithmic results is closer to your classification.

Without more information we cannot tell what these algorithms were doing. It may well be that they were just using different thresholds internally: Algo 1 decided that everything > 60% threshold is "positive", all < 30% "negative", all in between is "neutral". Algo 2 may have used 75%/25%.

By the way, Twitter messages are not really well-suited for sentiment analysis. They are too short and grammatically too messy.

You might look for emoticons inside the tweet-text only , that might work if you have lots+lots of tweets.

$\endgroup$
1
$\begingroup$

You'll need labeled data. It’s best if you have your own, but if you don’t, make sure you take your labeled data from the "Twitter" domain. This is important because tweets have a very specific style of writing, a specific lexicon, and use many emoticons. To help, here’s a database of 5K manually labeled tweets from Nick Sanders: http://www.sananalytics.com/lab/twitter-sentiment/

Run the labeled data through several models and calculate a metric. It could be a confusion matrix if you like. If you really don't need to have very precise classifications for all three classes (positive, neutral, negative), and if you need to identify, say, negative tweets, you can use metrics like precision and recall.

If you want more information about this, my colleagues and I have written a blog post describing how the performance of several models was compared in Twitter sentiment analysis: https://blog.griddynamics.com/selecting-training-evaluating-and-tuning-the-model-for-twitter-stream-sentiment-analysis-of-social-movie-review

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.