NLTK: Tuning LinearSVC classifier accuracy? - Looking for better approaches/advices

Question

Problem/Main objective/TLDR: Train a classifier, then feed it a random review and get the correspondent predicted review rating (number of stars from 1 to 5) - only 60% accuracy! :(

I have a big dataset with around 48000 tech product reviews (from many different writers and from different products - here this is not so important (?)) and corresponding ratings (1 to 5 stars) I randomly selected some reviews within each class:

1 star: 173 reviews (could not pick 1000 because there were 173)
2 stars: 1000 reviews
3 stars: 1000 reviews
4 stars: 1000 reviews
5 stars: 1000 reviews

Total: 4173 reviews - this data is organized in one file (all_reviews_labeled.txt) in tuple format, one review and rating for line:

(‘review text’, ‘x star’)
(‘review text’, ‘x star’)
(‘review text’, ‘x star’)
(‘review text’, ‘x star’)
…

My 1st “dummie” aproach was:

Tokenize review text
POS tagging

Get most frequent bigrams that folowing some POS tags rules for most frequent trigrams (I have seen this rules - using this POS patterns in “Automatic Star-rating Generation from Text Reviews” - pag.7 - paper from Chong-U Lim, Pablo Ortiz and Sang-Woo Jun):

for (w1,t1), (w2,t2), (w3,t3) in nltk.trigrams(text):
    if (t1 == 'JJ' or t1 == 'JJS' or t1 == 'JJR') and (t2 == 'NN' or t2 == 'NNS'):
        bi = unicode(w1 + ' ' + w2).encode('utf-8')
        bigrams.append(bi)
    elif (t1 == 'RB' or t1 == 'RBR' or t1 == 'RBS') and (t2 == 'JJ' or t2 == 'JJS' or t2 == 'JJR') and (t3 != 'NN' or t3 != 'NNS'):
        bi = unicode(w1 + ' ' + w2).encode('utf-8')
        bigrams.append(bi)
    elif (t1 == 'JJ' or t1 == 'JJS' or t1 == 'JJR') and (t2 == 'JJ' or t2 == 'JJS' or t2 == 'JJRS') and (t3 != 'NN' or t3 != 'NNS'):
        bi = unicode(w1 + ' ' + w2).encode('utf-8')
        bigrams.append(bi)
    elif (t1 == 'NN' or t1 == 'NNS') and (t2 == 'JJ' or t2 == 'JJS' or t2 == 'JJRS') and (t3 != 'NN' or t3 != 'NNS'):
        bi = unicode(w1 + ' ' + w2).encode('utf-8')
        bigrams.append(bi)
    elif (t1 == 'RB' or t1 == 'RBR' or t1 == 'RBS') and (t2 == 'VB' or t2 == 'VBD' or t2 == 'VBN' or t2 == 'VBG'):
        bi = unicode(w1 + ' ' + w2).encode('utf-8')
        bigrams.append(bi)
    elif (t1 == 'DT') and (t2 == 'JJ' or t2 == 'JJS' or t2 == 'JJRS'):
        bi = unicode(w1 + ' ' + w2).encode('utf-8')
        bigrams.append(bi)
    elif (t1 == 'VBZ') and (t2 == 'JJ' or t2 == 'JJS' or t2 == 'JJRS'):
        bi = unicode(w1 + ' ' + w2).encode('utf-8')
        bigrams.append(bi)
    else:
        continue

Extract features (here is where I have more doubts - should I only look for this two features?):

features={}
for bigram,freq in word_features:
    features['contains(%s)' % unicode(bigram).encode('utf-8')] = True
    features["count({})".format(unicode(bigram).encode('utf-8'))] = freq
return features

featuresets = [(review_features(review), rating) for (review, rating) in tuples_labeled_reviews]

Splits the training data into training size and testing size (90% training - 10% testing):

numtrain = int(len(tuples_labeled_reviews) * 90 / 100)
train_set, test_set = featuresets[:numtrain], featuresets[numtrain:]

Train SVMc:

classifier = nltk.classify.SklearnClassifier(LinearSVC())
classifier.train(train_set)

Evaluate the classifier:

errors = 0
correct = 0
for review, rating in test_set:
    tagged_rating = classifier.classify(review)
    if tagged_rating == rating:
        correct += 1
        print("Correct")
        print "Guess: ", tagged_rating
        print "Correct: ", rating
    else:
        errors += 1

So far I get only 60% accuracy… What can I do to improve my prediction results? Is something before, some text/reviews preprocessing (like removing stopwords/punctuation?) that is missing? Could you suggest me some other approaches? I am still a bit confused if is really a classification problem or a regression one... :/

Please simple explanations, or give me a link to “machine learning for dummies”, or be my mentor, I promise to learn fast! My background in machine learning/language processing/data mining is very light, I have played a couple of times with weka (Java), but now I need to stick with Python (nltk + scikit-learn)!

EDIT:

1. Now I am also extracting unigrams as features, unigrams POS-tagged as 'JJ', 'NN','VB' and 'RB'. It improved a little the accuracy to 65%.
2. I applied also do stemming and lemmatization in text before POS tagging. It improved the accuracy to +70%.

EDIT 2:

3. I have feed the classifier all my reviews, the 48000, split into 90% training and 10% testing and the accuracy was 91%.

4. Now I have 32000 new reviews (also labeled) and feed them all for testing and the mean accuracy was 62 % ... my confusion matrix is something like this image below (i divided by equal errors of +1/-1 star point, +2/-2, +3/-3 - because it is just an illustration):

What is happening? Why accuracy drops so much at 3 and 5 stars?

Answer 1

You might try looking into sentiment analysis. There was a kaggle competition on it, and you might find insight there.

Treating this as either a regression or a classification problem is fair. Also, it's important to judge your performance against the proper baselines.

Your feature space might not be rich enough for the classes to be linearly separable. You might do better using an SVM with a non-linear kernel.

It also appears you haven't scaled the counts of the bigrams, which is generally helpful for SVMs.

Another thought for an approach would be to apply LDA to the set of documents (reviews) and use the topics as your feature space (you'll have a topic vector per document).

Some places to get python LDA implementations:

gensim

Blei