3
$\begingroup$

As a part of my university project, I am researching/developing a sentiment analysis model where in I am trying to combine NLTK Vader (SentimentIntensityAnalyzer) results with a Machine Learning trained classifier for prediction of Sentiments on Twitter data.

Detailed description -

To explain what I am trying to do is - Combine Machine Learning classifier and NLTK Vader sentiment analysis to get better classification of tweets as Positive, Negative or Neutral.

What I have done is -

  1. Cleaned the data (Niek Sanders twitter corpus) and pre-processed the tweets which includes stop words removal, URL removal, User mention removal, remove # symbol from Hashtags, lower case conversion, stemmer processing, etc.

  2. Split into 80:20 training:test ratio

  3. Used TfidfVectorizer to create sparse matrix of features with TFIDF of words. The number of columns is equal to the number of words in clean data.

  4. Use this vector matrix to train and test the classifiers using scikit-learn.

Classifiers used - This same vector matrix is being used to train - KNN, Random forest, Naive Bayes, SVM, Artificial Neural Network and Convolutional Neural Network.

Now the main doubt arises when trying to combine NLTK Vader (SentimentIntensityAnalyzer results).

What I am doing is - From step three above, in the tfidf vector matrix I am adding 2 columns and I am adding the Positive and Negative polarity result for the tweet given by NLTK, so now the vector matrix has 2 new columns n+1 (total positive polarity of tweet by NLTK), n+2 (total negative polarity of tweet by NLTK) and it looks like -

    0      1     2     3     4     5     6     7  ......... |  n+1    |   n+2
------------------------------------------------------------|---------|----------
    0.4   0.3   0.4   0.1   0.5   0.3   0.2   0.4 ......... |  0.345  |  0.345
    0.5   0.3   0.2   0.8   0.3   0.6   0.4   0.5 ......... |  0.765  |  0.523
    0.6   0.4   0.1   0.7   0.8   0.8   0.2   0.2 ......... |  0.392  |  0.664
    0.2   0.9   0.7   0.4   0.9   0.9   0.8   0.5 ......... |  0.832  |  0.658
    0.9   0.5   0.9   0.7   0.3   0.2   0.2   0.5 ......... |  0.273  |  0.283
    0.5   0.2   0.2   0.7   0.2   0.1   0.6   0.6 ......... |  0.505  |  0.194
    0.4   0.3   0.2   0.3   0.3   0.9   0.5   0.5 ......... |  0.102  |  0.927
    0.1   0.8   0.1   0.2   0.1   0.5   0.2   0.7 ......... |  0.735  |  0.455

Question - So is it correct to add it this way?

I could also convert these polarities to binary values to mark if the overall sentiment is positive, negative or neutral using 0 and 1.

Note - I am representing 3 categories(positive, negative or neutral) using 2 columns to avoid dummy variable trap.

Though it has increased the overall accuracy of the classifiers by small margin of 1-2%.

But am I doing it right? If not, kindly let me know how can I collaborate the two mentioned above, i.e. NLTK Vader results with Machine Learning classifiers.

$\endgroup$
  • 1
    $\begingroup$ I'm answering you a year later, I hope this is still relevant to you. So, I came across your post as I'm facing the same problem at the moment. I'm researching on sentiment analysis for social media in Chinese. For my first baseline, I made my own implementation of VADER for Chinese with the goal to predict sentiment for Weibo. With some modifications it works reasonably well ~ 90% accuracy. I introduced some POS rules to make sentiment predictions, but unfortunately, Chinese language is a bit more complicated than English and there are not much good lexicons available. Therefore, I wish to im $\endgroup$ – Thiago Aug 10 '18 at 9:27
  • 1
    $\begingroup$ I am also doing the same thing for my uninversity research project. Can you guys please tell me how did you it? I am a beginner and I am still struggling with it. And so I would like to know if adding column in tfid matrix works well? $\endgroup$ – user64648 Dec 20 '18 at 12:40
0
$\begingroup$

Interesting approach, but the whole purpose of NLTK Vader is to have a pre-trained model. After all, NLTK Vader was manually (!) labeled.

I just tested Google vs. NLTK Vader on "I did not hate this movie" (negations are notoriously hard to catch for an algorithm) and NLTK Vader did much better than Google. NLTK Vader scored it kind of positive (0.45) while Google scored it negatively (-0.6).

Interesting ... :-)

strong text

enter image description here

Good luck!

$\endgroup$
0
$\begingroup$

I'll go through your questions one by one:


Question - So is it correct to add it this way?

It's a potentially good approach. A potential improvement (but that's a personal opinion) is to change the overall polarity scores with something else. I think that approach is too much "bag of words" (i.e. it looses the sequence information).

Since you are familiar with Neural Networks, I would suggest you to employ RNNs. Here's why: when you are working with natural language, sometimes the order of words is more important than the meaning of the word itself. Sometimes the order of words itself determines their meaning! That's why NLP achieves state-of-the-art performances when models that are able to understand sequences are employed (such as RNNs).


I could also convert these polarities to binary values to mark if the overall sentiment is positive, negative or neutral using 0 and 1.

That's a great point. In this way, you can train an RNN to read the sequence, and based on both word representations and their sequence, you can come to a classification.

The model I have in mind would start with recurrent layers (LSTM or GRU), that would learn a representation of the sequential information. Later, this information would be passed to dense layers, that can then perform the classification task.


Note - I am representing 3 categories(positive, negative or neutral) using 2 columns to avoid dummy variable trap.

If you are working with Neural Networks, I suggest you to use three output nodes with a softmax activation function (for the last layer).


But am I doing it right? If not, kindly let me know how can I collaborate the two mentioned above, i.e. NLTK Vader results with Machine Learning classifiers.

Your model can be good, I only suggested potential improvements.

Another thing I'd suggest is to use word embeddings, such as word2vec or Glove, instead of tf-idf representation of words. That is because Neural Networks love dense vectors, while they tend to work not so well on sparse vectors (such as tf-idf). You can find pre-trained datasets online just googling them.


Good luck with your task!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.