Naive Bayes for SA in Scikit Learn - how does it work

Question

Okay so i scrape data from the web on movie reviews. I also have already got my own 'dictionary' or 'lexicon' with words and their labels (1-poor, 2-ok, 3-good, 4-very good, 5-excellent).

SO the input are paragraphs of movie reviews and i use Scikit Learn Naive Bayes to evaluate the sentiment of each comment , which would be a paragraph. I would like to know how it works under the hood. I ASSUME it uses a bag of words concept.

So , I am describing my assumption about how Scikit Learn Naive Bayes works for SA. How close am i to the reality ?

I am neither a data scientist nor a statistician, but this is a summary of what i THINK happens in Naive Bayes algorithms for Sentiment Analysis, in Scikit Learn.

Training Phase

Lets assume i am using labels like 1,2,3,4,5 for each paragraph in the training set. Each paragraph is one unit for training as well as evaluation.

STEPS :-

1) Drop unwanted words like THE, BUT, AND and so on

2) Read the first word say 'BEACH', pick it's label from it's parent paragraph, say '5'. So attach 5 to BEACH and put it back in the bag.

3) So add up the number of times each word matched a given label. Same word 'BEACH' could occur with multiple labels across the input paragraphs. So keep count of each label for the given word. So maybe ['BEACH' - label 1 - 10], ['BEACH' - label 2 - 8] and so on.

4) After above step for all words, sum up the probability of getting a certain label for a given word. So "BEACH" may have 1/6 probability of label 1, 2/6 probability of 2, 1/6 probability of 3, 1/6 probability of 4 and 1/6 probability of 5. So these 5 probabilities for 'BEACH" are put back in the bag of words. Remove the Word-label-count entities from Step 3 , which were used to obtain these probabilities. So now each word can have a maximum of 5 probabilities in the bag.

Evaluating Sentiment Analysis - New Data

1) Now when we feed the real data, when it hits a word, it checks if it is found in its bag. If not it omits it

2) Say it finds BEACH in one comment. It checks highest probability for 'BEACH' is 2/6 for label 2. So it picks label 2 for the word BEACH. Similarly it picks corresponding labels with highest probability for all the words in bag matching the input.

3) Now it sums up all the labels for all words for the paragraph whose SA needs to be computed. Let us say they add up to 20. It has to convert this 20 to somewhere between 0 and 1. So it uses a logarithmic conversion which converts 20 to somewhere between 0 and 1. Let us say 0.75.

4) So 0.75 is the weight for this comment.

Is this how it works ? Thanks for any inputs.

Answer 1

Collecting your data

From the comments you state that you wish to classify comments into a label (1-poor, 2-fair, 3-ok, 4-good, 5-very good). Thus you will be training a model that maps a set of words (the paragraph) to a number which represents the rating. I will assume that you also have labels for some of the comments which we will call your training set.

I do not have access to your data but I have a similar dummy example which uses comments and tries to classify them as being a review of a hotel or a car rental. I think this example would suit this purpose nicely.

Pre-process the data

There's many techniques we can use for this purpose. For example some popula rones are bag-of-words, tf-idf, and n-grams. These techniques will take a comment and vectorize it into a set of features. The features which we will use are learned from the training set and then will be the same for new comments coming in. We will use bag-of-words for this example.

Bag-of-words

This method has a dictionary of words which we identify from our training set. We take all the unique words after stemming. We then count the number of times that each word shows up in the comment, that is the vector that represents a given instance. For example if our dictionary looks like

['car', 'book', 'test', 'wow']

And the comment was

Wow, this car is AMAZING!!!!!

The resulting vector would be

[1, 0, 0, 1]

We will repeat this vectorization process for each comment. We will then be left with a matrix where the rows represents each comment and the columns represent the frequency at which these words appear in the comment. We will call this matrix $X$ it is our dataset. Each of these instances will also have a label we will call that vector $Y$. We want to map each row in $X$ to it's label $Y$.

Assume this is the data you have for 40 instances in a training dataset for the cars/hotel comments. With it's corresponding label on the right. Label 0 is for cars and label 1 is for hotels.

The columns of the matrix $X$ are

['car', 'passenger', 'seat', 'drive', 'power', 'highway', 'purchase', 'hotel', 'room', 'night', 'staff', 'water', 'location']

Split the data

We will split the data in order to test the accuracy of our model

from sklearn.cross_validation import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3)

Apply Naive Bayes

from sklearn.naive_bayes import GaussianNB

clf = GaussianNB()
clf = clf.fit(X_train, y_train)
clf.score(X_test, y_test)

This gives a result of

1.0

Perfect classification!

---

How does it work?

Gaussian Naive Bayes assumes that each feature is described by a Gaussian distribution (you can pick other distributions which may be better suited for your data, such as multinomial). It will then assume that each feature is entirely independent of one another. Thus we can say that the probability of

$P(Y=0|X) = P(X_0|Y=0)P(X_0) * ... * P(X_n|Y=0)P(X_n)$

$P(Y=1|X) = P(X_0|Y=1)P(X_0) * ... * P(X_n|Y=1)P(X_n)$

for your $n$ features. The the one that is the largest is going to be our label. If $P(Y=1|X)$ is larger then we will say that the comment was that of a car. So to do this we need to train the parameters of our probability distributions $P(X_i|Y)$. To calculate this term we need to know the parameters of our distribution, in our case we are using a Gaussian distribution, thus we need the mean and the variance $\mathcal{N}(\mu, \theta)$.

We will build a dictionary of values for each label that contains the mean and variance for each feature. This is the training stage!

Homemade Naive Bayes

import csv
import random
import math
import pandas as pd
import numpy as np

class NaiveBayes(object):

    def __init__(self):
        self.groupClass = None
        self.stats = None

    def calculateGaussian(self, x, mean, std):    
        exponent = np.exp(-1*(np.power(x-mean,2)/(2*np.power(std,2))))
        std[std==0] = 0.00001
        return (1 / (np.sqrt(2*math.pi) * std)) * exponent

    def predict(self, x):
        probs = np.ones((len(x), len(self.stats)))

        for ix, instance in enumerate(x):
            for label_ix, label in enumerate(self.stats):
                probs[ix, int(label)] = probs[ix, int(label)] * \
                                      np.prod(self.calculateGaussian(instance, self.stats[label][0], self.stats[label][1]))
        return np.argmax(probs, 1)

    def score(self, x, y):
        pred = self.predict(x)
        return np.sum(1-np.abs(y - pred))/len(x)

    def train(self, x, y):
        self.splitClasses(x, y)
        self.getStats()
        pass

    def splitClasses(self, x, y):
        groupClass = {}

        for instance, label in zip(x, y):
            if not label in groupClass:
                groupClass.update({label: [instance]})
            else:
                groupClass[label].append(instance)
        self.groupClass = groupClass

    def getStats(self):
        stats = {}

        for label in self.groupClass:
            mean = np.mean(np.asarray(self.groupClass[label]), 0)
            std = np.std(np.asarray(self.groupClass[label]), 0)
            stats.update({label: [mean, std]})
        self.stats = stats

clf = NaiveBayes()
clf.train(X_train, y_train)
clf.score(X_test, y_test)

Once again we get an accuracy of 1.0 on the testing set!

Answer 2

In Naive Bayes, you need two values :

• The prior of each class : $p(c_j)$ which is the proportion of each class in the training set.
• The conditional probability of each term $i$ from a document regarding to each class : $p(w_i|c_j) = \frac{count(w_i, c_j)}{\sum\limits_{w\in V}count(w, c_j)}$ which corresponds to the fraction of times the word $w_i$ appears among all words in documents of class $c_j$ ($V$ is the vocabulary).

The classification rule being $\underset{c_j}{argmax} \; p(c_j)p(w|c_j)$ with $p(w|c_j) = \prod\limits_{w_i}p(w_i|c_j)$ thanks to the independence assumption.

To prevent underflow errors, we often use the log : $log \;\;p(w|c_j) = \sum\limits_{w_i}log\;\;p(w_i|c_j)$

Therefore, the final word is : to classify a document (or paragraph or whatever piece of text), you take each word $i$ from it, and for each class $j$, see the fraction of times it appears in it (in documents from class $j$), in the training data (which is a probability), and add up the log. It will give you $j$ probabilities (one for each class), and you take the class corresponding to the maximum.

What is not clear in your question is the relation between labels and sentiment analysis.

See this for a detailed explanation and examples.