Okay so i scrape data from the web on movie reviews. I also have already got my own 'dictionary' or 'lexicon' with words and their labels (1-poor, 2-ok, 3-good, 4-very good, 5-excellent).
SO the input are paragraphs of movie reviews and i use Scikit Learn Naive Bayes to evaluate the sentiment of each comment , which would be a paragraph. I would like to know how it works under the hood. I ASSUME it uses a bag of words concept.
So , I am describing my assumption about how Scikit Learn Naive Bayes works for SA. How close am i to the reality ?
I am neither a data scientist nor a statistician, but this is a summary of what i THINK happens in Naive Bayes algorithms for Sentiment Analysis, in Scikit Learn.
Lets assume i am using labels like 1,2,3,4,5 for each paragraph in the training set. Each paragraph is one unit for training as well as evaluation.
1) Drop unwanted words like THE, BUT, AND and so on
2) Read the first word say 'BEACH', pick it's label from it's parent paragraph, say '5'. So attach 5 to BEACH and put it back in the bag.
3) So add up the number of times each word matched a given label. Same word 'BEACH' could occur with multiple labels across the input paragraphs. So keep count of each label for the given word. So maybe ['BEACH' - label 1 - 10], ['BEACH' - label 2 - 8] and so on.
4) After above step for all words, sum up the probability of getting a certain label for a given word. So "BEACH" may have 1/6 probability of label 1, 2/6 probability of 2, 1/6 probability of 3, 1/6 probability of 4 and 1/6 probability of 5. So these 5 probabilities for 'BEACH" are put back in the bag of words. Remove the Word-label-count entities from Step 3 , which were used to obtain these probabilities. So now each word can have a maximum of 5 probabilities in the bag.
Evaluating Sentiment Analysis - New Data
1) Now when we feed the real data, when it hits a word, it checks if it is found in its bag. If not it omits it
2) Say it finds BEACH in one comment. It checks highest probability for 'BEACH' is 2/6 for label 2. So it picks label 2 for the word BEACH. Similarly it picks corresponding labels with highest probability for all the words in bag matching the input.
3) Now it sums up all the labels for all words for the paragraph whose SA needs to be computed. Let us say they add up to 20. It has to convert this 20 to somewhere between 0 and 1. So it uses a logarithmic conversion which converts 20 to somewhere between 0 and 1. Let us say 0.75.
4) So 0.75 is the weight for this comment.
Is this how it works ? Thanks for any inputs.