# Naive Bayes for SA in Scikit Learn - how does it work

Okay so i scrape data from the web on movie reviews. I also have already got my own 'dictionary' or 'lexicon' with words and their labels (1-poor, 2-ok, 3-good, 4-very good, 5-excellent).

SO the input are paragraphs of movie reviews and i use Scikit Learn Naive Bayes to evaluate the sentiment of each comment , which would be a paragraph. I would like to know how it works under the hood. I ASSUME it uses a bag of words concept.

So , I am describing my assumption about how Scikit Learn Naive Bayes works for SA. How close am i to the reality ?

I am neither a data scientist nor a statistician, but this is a summary of what i THINK happens in Naive Bayes algorithms for Sentiment Analysis, in Scikit Learn.

Training Phase

Lets assume i am using labels like 1,2,3,4,5 for each paragraph in the training set. Each paragraph is one unit for training as well as evaluation.

STEPS :-

1) Drop unwanted words like THE, BUT, AND and so on

2) Read the first word say 'BEACH', pick it's label from it's parent paragraph, say '5'. So attach 5 to BEACH and put it back in the bag.

3) So add up the number of times each word matched a given label. Same word 'BEACH' could occur with multiple labels across the input paragraphs. So keep count of each label for the given word. So maybe ['BEACH' - label 1 - 10], ['BEACH' - label 2 - 8] and so on.

4) After above step for all words, sum up the probability of getting a certain label for a given word. So "BEACH" may have 1/6 probability of label 1, 2/6 probability of 2, 1/6 probability of 3, 1/6 probability of 4 and 1/6 probability of 5. So these 5 probabilities for 'BEACH" are put back in the bag of words. Remove the Word-label-count entities from Step 3 , which were used to obtain these probabilities. So now each word can have a maximum of 5 probabilities in the bag.

Evaluating Sentiment Analysis - New Data

1) Now when we feed the real data, when it hits a word, it checks if it is found in its bag. If not it omits it

2) Say it finds BEACH in one comment. It checks highest probability for 'BEACH' is 2/6 for label 2. So it picks label 2 for the word BEACH. Similarly it picks corresponding labels with highest probability for all the words in bag matching the input.

3) Now it sums up all the labels for all words for the paragraph whose SA needs to be computed. Let us say they add up to 20. It has to convert this 20 to somewhere between 0 and 1. So it uses a logarithmic conversion which converts 20 to somewhere between 0 and 1. Let us say 0.75.

4) So 0.75 is the weight for this comment.

Is this how it works ? Thanks for any inputs.

• To understand an algorithm it's always best to choose the simplest case to remove all the nitty gritty stuff. Naive Bayes is the same regardless of what data you use. However, the majority of your description focuses on a bag-of-words approach and obfuscates the Naive Bayes question. – JahKnows May 30 '18 at 10:03
• Okay, i wanted to know in context of Scikit Learn in Python . – Chakra May 30 '18 at 10:05
• Can you give us a sample of the data and I'll write a small how-to? – JahKnows May 30 '18 at 10:09
• Okay our team is already using NaiveBayes from Scikit Learn to do SA. So I wanted to know how it works under the hood. – Chakra May 30 '18 at 10:11
• Naive Bayes does not do any bag-of-words or any pre-processing, you need to do all these steps separately. Naive Bayes is simply a statistical model which can be used to classify data from a novel set based on the parameters it has learned from the training set. – JahKnows May 30 '18 at 10:25

From the comments you state that you wish to classify comments into a label (1-poor, 2-fair, 3-ok, 4-good, 5-very good). Thus you will be training a model that maps a set of words (the paragraph) to a number which represents the rating. I will assume that you also have labels for some of the comments which we will call your training set.

I do not have access to your data but I have a similar dummy example which uses comments and tries to classify them as being a review of a hotel or a car rental. I think this example would suit this purpose nicely.

# Pre-process the data

There's many techniques we can use for this purpose. For example some popula rones are bag-of-words, tf-idf, and n-grams. These techniques will take a comment and vectorize it into a set of features. The features which we will use are learned from the training set and then will be the same for new comments coming in. We will use bag-of-words for this example.

## Bag-of-words

This method has a dictionary of words which we identify from our training set. We take all the unique words after stemming. We then count the number of times that each word shows up in the comment, that is the vector that represents a given instance. For example if our dictionary looks like

['car', 'book', 'test', 'wow']

And the comment was

Wow, this car is AMAZING!!!!!

The resulting vector would be

[1, 0, 0, 1]

We will repeat this vectorization process for each comment. We will then be left with a matrix where the rows represents each comment and the columns represent the frequency at which these words appear in the comment. We will call this matrix $X$ it is our dataset. Each of these instances will also have a label we will call that vector $Y$. We want to map each row in $X$ to it's label $Y$.

Assume this is the data you have for 40 instances in a training dataset for the cars/hotel comments. With it's corresponding label on the right. Label 0 is for cars and label 1 is for hotels.

The columns of the matrix $X$ are

['car', 'passenger', 'seat', 'drive', 'power', 'highway', 'purchase', 'hotel', 'room', 'night', 'staff', 'water', 'location']

# Split the data

We will split the data in order to test the accuracy of our model

from sklearn.cross_validation import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3)

# Apply Naive Bayes

from sklearn.naive_bayes import GaussianNB

clf = GaussianNB()
clf = clf.fit(X_train, y_train)
clf.score(X_test, y_test)

This gives a result of

1.0

Perfect classification!

# How does it work?

Gaussian Naive Bayes assumes that each feature is described by a Gaussian distribution (you can pick other distributions which may be better suited for your data, such as multinomial). It will then assume that each feature is entirely independent of one another. Thus we can say that the probability of

$P(Y=0|X) = P(X_0|Y=0)P(X_0) * ... * P(X_n|Y=0)P(X_n)$

$P(Y=1|X) = P(X_0|Y=1)P(X_0) * ... * P(X_n|Y=1)P(X_n)$

for your $n$ features. The the one that is the largest is going to be our label. If $P(Y=1|X)$ is larger then we will say that the comment was that of a car. So to do this we need to train the parameters of our probability distributions $P(X_i|Y)$. To calculate this term we need to know the parameters of our distribution, in our case we are using a Gaussian distribution, thus we need the mean and the variance $\mathcal{N}(\mu, \theta)$.

We will build a dictionary of values for each label that contains the mean and variance for each feature. This is the training stage!

import csv
import random
import math
import pandas as pd
import numpy as np

class NaiveBayes(object):

def __init__(self):
self.groupClass = None
self.stats = None

def calculateGaussian(self, x, mean, std):
exponent = np.exp(-1*(np.power(x-mean,2)/(2*np.power(std,2))))
std[std==0] = 0.00001
return (1 / (np.sqrt(2*math.pi) * std)) * exponent

def predict(self, x):
probs = np.ones((len(x), len(self.stats)))

for ix, instance in enumerate(x):
for label_ix, label in enumerate(self.stats):
probs[ix, int(label)] = probs[ix, int(label)] * \
np.prod(self.calculateGaussian(instance, self.stats[label][0], self.stats[label][1]))
return np.argmax(probs, 1)

def score(self, x, y):
pred = self.predict(x)
return np.sum(1-np.abs(y - pred))/len(x)

def train(self, x, y):
self.splitClasses(x, y)
self.getStats()
pass

def splitClasses(self, x, y):
groupClass = {}

for instance, label in zip(x, y):
if not label in groupClass:
groupClass.update({label: [instance]})
else:
groupClass[label].append(instance)
self.groupClass = groupClass

def getStats(self):
stats = {}

for label in self.groupClass:
mean = np.mean(np.asarray(self.groupClass[label]), 0)
std = np.std(np.asarray(self.groupClass[label]), 0)
stats.update({label: [mean, std]})
self.stats = stats

clf = NaiveBayes()
clf.train(X_train, y_train)
clf.score(X_test, y_test)

Once again we get an accuracy of 1.0 on the testing set!

• I'll append some homemade Naive Baye's code soon so you can see what is happening under the hood. – JahKnows May 30 '18 at 10:57
• Very detailed and clear explanation, thank you and +1. However, I am wondering what the first graphic illustration refers to, as it doesn't really match the description before it, there is no label 0 and 1 on the right of anything there. Can you please explain? – OMan May 30 '18 at 14:12
• @OMan, there's two matrices. The one on the left is the instances as rows and features as columns. The intensity of the pixels is the word frequency. On the right side, the purple pixels are labels 0, and the yellow pixels are label 1. Hotels are 0, cars are 1. – JahKnows May 30 '18 at 15:24

In Naive Bayes, you need two values :

• The prior of each class : $p(c_j)$ which is the proportion of each class in the training set.
• The conditional probability of each term $i$ from a document regarding to each class : $p(w_i|c_j) = \frac{count(w_i, c_j)}{\sum\limits_{w\in V}count(w, c_j)}$ which corresponds to the fraction of times the word $w_i$ appears among all words in documents of class $c_j$ ($V$ is the vocabulary).

The classification rule being $\underset{c_j}{argmax} \; p(c_j)p(w|c_j)$ with $p(w|c_j) = \prod\limits_{w_i}p(w_i|c_j)$ thanks to the independence assumption.

To prevent underflow errors, we often use the log : $log \;\;p(w|c_j) = \sum\limits_{w_i}log\;\;p(w_i|c_j)$

Therefore, the final word is : to classify a document (or paragraph or whatever piece of text), you take each word $i$ from it, and for each class $j$, see the fraction of times it appears in it (in documents from class $j$), in the training data (which is a probability), and add up the log. It will give you $j$ probabilities (one for each class), and you take the class corresponding to the maximum.

What is not clear in your question is the relation between labels and sentiment analysis.

See this for a detailed explanation and examples.