0
$\begingroup$

I am referring http://www.nltk.org/book/ch06.html for generating a movie review classifier. It considers all words (Nouns, adjectives, verbs..) as part of feature set. I am trying to build a classifier that considers only the verbs and assesses if the movie review is positive or negative.


Please explain if this approach is better, if yes how can it be improved else what other parts of speech tags need to be included to improve the feature set.
Please refer to the following code
1)Import nltk and corpus
2)Understand the categories
3)Create a list of "documents" where each document consists of words that are not in stopwords.words() and filtered from list of "punctuations" as well.
4)Shuffle the documents and generate a list called "all_words" that contains all the words appearing in movie_reviews minus stopwords and punctuations.
5)Create a Freq Dist of "all_words"
6)Create a list of "verbs" by looking at the pos_tag of each word appearing in "all_words"
7)Create a feature dictionary for each document in the "documents" list created in step 3, such that the dictionary will contain keys corresponding to words in "verb" list and a boolean to show if that verb appeared in the document under consideration. This is handled by documentFeature().
8)Create a naive bayes classifier instance and train and compute test accuracy.

from nltk.corpus import movie_reviews
from nltk.corpus import stopwords
from nltk.tokenize import RegexpTokenizer
import nltk
movie_reviews.categories()
#['pos','neg']
#the regextokenier is used to tokenize the words    
tokenizer=RegexpTokenizer(r'\w+')
#creating documents based on filtration of stopwords for each review and running a tokenizer on each document
documents=[(tokenizer.tokenize(' '.join(set(i for i in movie_reviews.words(fileid))-set(stopwords.words()))),category) 
           for category in movie_reviews.categories()
           for fileid in movie_reviews.fileids() 
          ]
import random
random.shuffle(documents)
#each document contains words that are not in stopwords and punctuations.
for i in documents[:5]:
    temp=nltk.FreqDist([j.lower() for j in i[0]])
    print(temp.most_common(5),i[1]) 
#output of 5 documents
#[('vampires', 1), ('clever', 1), ('interesting', 1), ('sunlight', 1), ('partners', 1)] neg
#[('family', 1), ('nino', 1), ('friends', 1), ('acting', 1), ('higher', 1)] pos
#[('inconsistent', 1), ('eye', 1), ('yes', 1), ('interesting', 1), ('praise', 1)] neg
#[('acting', 1), ('science', 1), ('bucks', 1), ('huge', 1), ('terrific', 1)] pos
#[('acting', 1), ('shielded', 1), ('somewhere', 1), ('think', 1), ('touched', 1)] neg

#generate a list called 'all_words' that contains all the set of words that have appeared so far
all_words=tokenizer.tokenize(' '.join(set(i for i in movie_reviews.words())-set(stopwords.words())))
freqdist=nltk.FreqDist(all_words)

#create a list of all verbs for each word appearing in 'all_words'
verb=[]
pos_=nltk.pos_tag(all_words)
#print([i[1] for i in pos_])
for i in pos_:
    if i[1] in ['VB','VBG','VBN','VBZ','VBD','VBP']:
        verb.append(i[0])

#document - feature set, build a dictionary of verbs for each document
def documentFeature(document):
    feature={}
    for i in verb:
        feature['contains({0})'.format(i)]=(i in document)
    return feature    
#build a naive bayes classifier
featureSet=[(documentFeature(d),c) for d,c in documents]
trainSet,testSet=featureSet[100:], featureSet[:100]
classifier=nltk.NaiveBayesClassifier.train(trainSet)

print(nltk.classify.accuracy(classifier, testSet))
#0.03 a very poor accuracy on the testset

Currently I am getting 0.03 accuracy, please help me with improving the accuracy.

$\endgroup$
  • $\begingroup$ 0.03% accuracy seems way too low even if verbs turn out to be a very bad feature. Maybe you have some bug or you are reading too few documents? $\endgroup$ – Miguel Dec 30 '17 at 11:52
  • $\begingroup$ Two questions: 1. Why Naive Bayes? 2. Why just the verbs? Starting out with these limitations is basically a form of researcher bias. Why not maintain an open mind to which models will work best with any number of possible inputs? Besides, for NLP there are many, many more algorithms that work better than Naive Bayes. $\endgroup$ – I_Play_With_Data Feb 28 '18 at 14:58
  • $\begingroup$ Can you please provide a link to the data so that we can help you? $\endgroup$ – JahKnows Oct 27 '18 at 2:14
0
$\begingroup$

Without reading your code (sorry!) I can suggest that you drop the verbs-only approach and use a word embedding to account for word similarity. Your feature dictionary is bound to find very few "hits" in the documents, unless you have a humongous dataset, and even then YMMV because you are just using verbs. Also: what's the justifications for using verbs? "I liked" and "I didn't like" both contain the verb "to like" and they mean opposite things. Same goes for hate (because you only have binary classification: no nuances) and so on.

A popular python implementation of word2vec is gensim, but you could use that of tensorflow or some other embedding like the (allegedly superior) conceptnet numberbatch. If you want to summarize whole documents into numbers you can try doc2vec (aka paragraph2vec, paper here), also available in gensim, tensorflow, etc.

You will need a big dataset. Once you have this up and running you can try other classifiers.

$\endgroup$
  • $\begingroup$ Would it make sense to acquire phrases of type 'I liked this',(3 words with verb in the middle) from corpus using nltk and create word vectors using gensim? $\endgroup$ – Ashwin V Dec 30 '17 at 13:37
  • $\begingroup$ I don't know... Word2Vec already captures information about surrounding words by design. It's actually what it's about. Also, it's not clear what you would do with the word vectors then (they are computed beforehand): averaging wouldn't make much sense for instance. I suggest you have a quick look at doc2vec to see how it works. It might give you some ideas. On the other hand, every test you make, gets you closer to understanding things and provides baselines against which you can benchmark better algorithms later :) $\endgroup$ – Miguel Dec 30 '17 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.