2
$\begingroup$

Problem/Main objective/TLDR: Train a classifier, then feed it a random review and get the correspondent predicted review rating (number of stars from 1 to 5) - only 60% accuracy! :(

I have a big dataset with around 48000 tech product reviews (from many different writers and from different products - here this is not so important (?)) and corresponding ratings (1 to 5 stars) I randomly selected some reviews within each class:

1 star: 173 reviews (could not pick 1000 because there were 173)
2 stars: 1000 reviews
3 stars: 1000 reviews
4 stars: 1000 reviews
5 stars: 1000 reviews

Total: 4173 reviews - this data is organized in one file (all_reviews_labeled.txt) in tuple format, one review and rating for line:

(‘review text’, ‘x star’)
(‘review text’, ‘x star’)
(‘review text’, ‘x star’)
(‘review text’, ‘x star’)
…

My 1st “dummie” aproach was:

Tokenize review text
POS tagging

Get most frequent bigrams that folowing some POS tags rules for most frequent trigrams (I have seen this rules - using this POS patterns in “Automatic Star-rating Generation from Text Reviews” - pag.7 - paper from Chong-U Lim, Pablo Ortiz and Sang-Woo Jun):

for (w1,t1), (w2,t2), (w3,t3) in nltk.trigrams(text):
    if (t1 == 'JJ' or t1 == 'JJS' or t1 == 'JJR') and (t2 == 'NN' or t2 == 'NNS'):
        bi = unicode(w1 + ' ' + w2).encode('utf-8')
        bigrams.append(bi)
    elif (t1 == 'RB' or t1 == 'RBR' or t1 == 'RBS') and (t2 == 'JJ' or t2 == 'JJS' or t2 == 'JJR') and (t3 != 'NN' or t3 != 'NNS'):
        bi = unicode(w1 + ' ' + w2).encode('utf-8')
        bigrams.append(bi)
    elif (t1 == 'JJ' or t1 == 'JJS' or t1 == 'JJR') and (t2 == 'JJ' or t2 == 'JJS' or t2 == 'JJRS') and (t3 != 'NN' or t3 != 'NNS'):
        bi = unicode(w1 + ' ' + w2).encode('utf-8')
        bigrams.append(bi)
    elif (t1 == 'NN' or t1 == 'NNS') and (t2 == 'JJ' or t2 == 'JJS' or t2 == 'JJRS') and (t3 != 'NN' or t3 != 'NNS'):
        bi = unicode(w1 + ' ' + w2).encode('utf-8')
        bigrams.append(bi)
    elif (t1 == 'RB' or t1 == 'RBR' or t1 == 'RBS') and (t2 == 'VB' or t2 == 'VBD' or t2 == 'VBN' or t2 == 'VBG'):
        bi = unicode(w1 + ' ' + w2).encode('utf-8')
        bigrams.append(bi)
    elif (t1 == 'DT') and (t2 == 'JJ' or t2 == 'JJS' or t2 == 'JJRS'):
        bi = unicode(w1 + ' ' + w2).encode('utf-8')
        bigrams.append(bi)
    elif (t1 == 'VBZ') and (t2 == 'JJ' or t2 == 'JJS' or t2 == 'JJRS'):
        bi = unicode(w1 + ' ' + w2).encode('utf-8')
        bigrams.append(bi)
    else:
        continue

Extract features (here is where I have more doubts - should I only look for this two features?):

features={}
for bigram,freq in word_features:
    features['contains(%s)' % unicode(bigram).encode('utf-8')] = True
    features["count({})".format(unicode(bigram).encode('utf-8'))] = freq
return features

featuresets = [(review_features(review), rating) for (review, rating) in tuples_labeled_reviews]

Splits the training data into training size and testing size (90% training - 10% testing):

numtrain = int(len(tuples_labeled_reviews) * 90 / 100)
train_set, test_set = featuresets[:numtrain], featuresets[numtrain:]

Train SVMc:

classifier = nltk.classify.SklearnClassifier(LinearSVC())
classifier.train(train_set)

Evaluate the classifier:

errors = 0
correct = 0
for review, rating in test_set:
    tagged_rating = classifier.classify(review)
    if tagged_rating == rating:
        correct += 1
        print("Correct")
        print "Guess: ", tagged_rating
        print "Correct: ", rating
    else:
        errors += 1

So far I get only 60% accuracy… What can I do to improve my prediction results? Is something before, some text/reviews preprocessing (like removing stopwords/punctuation?) that is missing? Could you suggest me some other approaches? I am still a bit confused if is really a classification problem or a regression one... :/

Please simple explanations, or give me a link to “machine learning for dummies”, or be my mentor, I promise to learn fast! My background in machine learning/language processing/data mining is very light, I have played a couple of times with weka (Java), but now I need to stick with Python (nltk + scikit-learn)!

EDIT:

  1. Now I am also extracting unigrams as features, unigrams POS-tagged as 'JJ', 'NN','VB' and 'RB'. It improved a little the accuracy to 65%.
  2. I applied also do stemming and lemmatization in text before POS tagging. It improved the accuracy to +70%.

EDIT 2:

  1. I have feed the classifier all my reviews, the 48000, split into 90% training and 10% testing and the accuracy was 91%.

  2. Now I have 32000 new reviews (also labeled) and feed them all for testing and the mean accuracy was 62 % ... my confusion matrix is something like this image below (i divided by equal errors of +1/-1 star point, +2/-2, +3/-3 - because it is just an illustration):

enter image description here What is happening? Why accuracy drops so much at 3 and 5 stars?

$\endgroup$
3
$\begingroup$

You might try looking into sentiment analysis. There was a kaggle competition on it, and you might find insight there.

Treating this as either a regression or a classification problem is fair. Also, it's important to judge your performance against the proper baselines.

Your feature space might not be rich enough for the classes to be linearly separable. You might do better using an SVM with a non-linear kernel.

It also appears you haven't scaled the counts of the bigrams, which is generally helpful for SVMs.

Another thought for an approach would be to apply LDA to the set of documents (reviews) and use the topics as your feature space (you'll have a topic vector per document).

Some places to get python LDA implementations:

gensim

Blei

$\endgroup$
  • $\begingroup$ thanks so much for your answer. I don't see how having a topic per document would help to predict rating... but I will have a look on LDA later. I will try the SVM with a non-linear kernel. thanks $\endgroup$ – Inês Martins Sep 3 '15 at 15:57
  • $\begingroup$ LDA seeks to cluster tokens (either words or n-grams) that co-occur in documents. This allows you to learn a lower-dimensional input than, say, word/n-gram frequency. If your initial intuition holds true (that the same n-grams will occur in reviews of the same sentiment), then these n-grams will cluster to topics that reflect that sentiment. Though I haven't seen this implemented in literature, I believe that labeling your documents with a "sentiment token" would help your topics converge. $\endgroup$ – jamesmf Sep 3 '15 at 16:36
  • $\begingroup$ With SVM with a non-linear kernel (so far I tried the default 'rbf') i only get 20% accuracy. I will try now with kernel "poly" $\endgroup$ – Inês Martins Sep 4 '15 at 13:21
  • $\begingroup$ with poly got the same accuracy: around 20% $\endgroup$ – Inês Martins Sep 7 '15 at 8:36
  • $\begingroup$ I have used all the dataset, the 48000 reviews, 90% training - 10% testing, with LinearSVC, looking at bigrams and adjectives only and get an accuracy of 93%, is it normal? also verified with 10 fold cross validation and mean accuracy is 99.6%... $\endgroup$ – Inês Martins Sep 8 '15 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.