You might try looking into [sentiment analysis][1]. There was a [kaggle competition][2] on it, and you might find insight there. Treating this as either a regression or a classification problem is fair. Your feature space might not be rich enough for the classes to be linearly separable. You might do better using an [SVM with a non-linear kernel][3]. It also appears you haven't [scaled][4] the counts of the bigrams, which is generally helpful for SVMs. Another thought for an approach would be to apply [LDA][5] to the set of documents (reviews) and use the topics as your feature space (you'll have a topic vector per document). Some places to get python LDA implementations: [gensim][6] [Blei][7] [1]: https://en.wikipedia.org/wiki/Sentiment_analysis [2]: https://www.kaggle.com/c/sentiment-analysis-on-movie-reviews [3]: http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html [4]: https://en.wikipedia.org/wiki/Feature_scaling [5]: https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation [6]: https://pypi.python.org/pypi/gensim [7]: https://www.cs.princeton.edu/~blei/topicmodeling.html