Combining Machine Learning classifier with NLTK Vader for Sentiment Analysis

Question

As a part of my university project, I am researching/developing a sentiment analysis model wherein I am trying to combine NLTK Vader (SentimentIntensityAnalyzer) results with a Machine Learning trained classifier for prediction of Sentiments on Twitter data.

Detailed description -

To explain what I am trying to do is - Combine Machine Learning classifier and NLTK Vader sentiment analysis to get the better classification of tweets as Positive, Negative or Neutral.

What I have done is -

1. Cleaned the data (Niek Sanders twitter corpus) and pre-processed the tweets which include stop words removal, URL removal, User mention removal, remove the # symbol from Hashtags, lower case conversion, stemmer processing, etc.

2. Split into 80:20 training: test ratio

3. Used TfidfVectorizer to create a sparse matrix of features with TFIDF of words. The number of columns is equal to the number of words in clean data.

4. Use this vector matrix to train and test the classifiers using scikit-learn.

Classifiers used - This same vector matrix is being used to train - KNN, Random forest, Naive Bayes, SVM, Artificial Neural Network and Convolutional Neural Network.

Now the main doubt arises when trying to combine NLTK Vader (SentimentIntensityAnalyzer results).

What I am doing is - From step three above, in the tfidf vector-matrix I am adding 2 columns and I am adding the Positive and Negative polarity result for the tweet given by NLTK, so now the vector-matrix has 2 new columns n+1 (total positive polarity of tweet by NLTK), n+2 (total negative polarity of tweet by NLTK) and it looks like -

    0      1     2     3     4     5     6     7  ......... |  n+1    |   n+2
------------------------------------------------------------|---------|----------
    0.4   0.3   0.4   0.1   0.5   0.3   0.2   0.4 ......... |  0.345  |  0.345
    0.5   0.3   0.2   0.8   0.3   0.6   0.4   0.5 ......... |  0.765  |  0.523
    0.6   0.4   0.1   0.7   0.8   0.8   0.2   0.2 ......... |  0.392  |  0.664
    0.2   0.9   0.7   0.4   0.9   0.9   0.8   0.5 ......... |  0.832  |  0.658
    0.9   0.5   0.9   0.7   0.3   0.2   0.2   0.5 ......... |  0.273  |  0.283
    0.5   0.2   0.2   0.7   0.2   0.1   0.6   0.6 ......... |  0.505  |  0.194
    0.4   0.3   0.2   0.3   0.3   0.9   0.5   0.5 ......... |  0.102  |  0.927
    0.1   0.8   0.1   0.2   0.1   0.5   0.2   0.7 ......... |  0.735  |  0.455

Question - So is it correct to add it this way?

I could also convert these polarities to binary values to mark if the overall sentiment is positive, negative or neutral using 0 and 1.

Note - I am representing 3 categories(positive, negative or neutral) using 2 columns to avoid dummy variable trap.

Though it has increased the overall accuracy of the classifiers by small margin of 1-2%.

But am I doing it right? If not, kindly let me know how can I collaborate the two mentioned above, i.e. NLTK Vader results with Machine Learning classifiers.

Answer 1

Interesting approach, but the whole purpose of NLTK Vader is to have a pre-trained model. After all, NLTK Vader was manually (!) labeled.

I just tested Google vs. NLTK Vader on "I did not hate this movie" (negations are notoriously hard to catch for an algorithm) and NLTK Vader did much better than Google. NLTK Vader scored it kind of positive (0.45) while Google scored it negatively (-0.6).

Interesting ... :-)

strong text

Good luck!

Answer 2

I'll go through your questions one by one:

---

Question - So is it correct to add it this way?

It's a potentially good approach. A potential improvement (but that's a personal opinion) is to change the overall polarity scores with something else. I think that approach is too much "bag of words" (i.e. it looses the sequence information).

Since you are familiar with Neural Networks, I would suggest you to employ RNNs. Here's why: when you are working with natural language, sometimes the order of words is more important than the meaning of the word itself. Sometimes the order of words itself determines their meaning! That's why NLP achieves state-of-the-art performances when models that are able to understand sequences are employed (such as RNNs).

---

I could also convert these polarities to binary values to mark if the overall sentiment is positive, negative or neutral using 0 and 1.

That's a great point. In this way, you can train an RNN to read the sequence, and based on both word representations and their sequence, you can come to a classification.

The model I have in mind would start with recurrent layers (LSTM or GRU), that would learn a representation of the sequential information. Later, this information would be passed to dense layers, that can then perform the classification task.

---

Note - I am representing 3 categories(positive, negative or neutral) using 2 columns to avoid dummy variable trap.

If you are working with Neural Networks, I suggest you to use three output nodes with a softmax activation function (for the last layer).

---

But am I doing it right? If not, kindly let me know how can I collaborate the two mentioned above, i.e. NLTK Vader results with Machine Learning classifiers.

Your model can be good, I only suggested potential improvements.

Another thing I'd suggest is to use word embeddings, such as word2vec or Glove, instead of tf-idf representation of words. That is because Neural Networks love dense vectors, while they tend to work not so well on sparse vectors (such as tf-idf). You can find pre-trained datasets online just googling them.

---

Good luck with your task!