6
$\begingroup$

I am currently working on sentiment analysis using Python. I wanted to find whether reviews given for a movie is positive or negative based on sentiment analysis. I have found a training dataset as provided in the below Link:

https://inclass.kaggle.com/c/si650winter11/data

This dataset have reviews and a score with 1 indicating review is positive and 0 indicating movie review is negative, but it has less number of records. I have a test dataset which I will predicting based on training set. My test dataset has complex and long words for which my python ML model sometimes gives positive result for a negative reviews (returning result as 1 for negative review). I am looking for better dataset to train my model, so that my model can predict well. Could you please advise me any good/large and effective training dataset to use for this scenario? If you could share any links that would be great. The training data can be in format as 1 for positive reviews and 0 for negative reviews or even polarity like pos for positive reviews or neg for negative reviews.

$\endgroup$
  • $\begingroup$ how different are the sizes of your training and testing datasets? After building your model, are you using cross validation? $\endgroup$ – Bach Apr 15 '16 at 17:05
7
$\begingroup$

You can use the SAR14 dataset of 234K IMDb movie reviews. The construction of the SAR14 dataset is detailed in the paper "Sentiment Classification on Polarity Reviews: An Empirical Study Using Rating-based Features".

$\endgroup$
0
$\begingroup$

~7000 sample data entries is definitely not enough considering that a more-or-less reliable dictionary of sentiment-loaded words consists of several thousand words each for both positive and negative sentiments. Basically, when you train the model you build such a dictionary in some sense.

There is, however, an existing training/test dataset consisting of 50000 reviews which is a bit better than what you have - http://ai.stanford.edu/~amaas/data/sentiment/

At the same time, while the amount of training sample data contributes to the quality of the classifier, it's also important that the style and dictionary used by authors of texts in the training set was similar to your test texts. In addition, text processing tricks like stemming may increase the training efficiency.

For more information, you might want to look at this blog posts several colleagues and I wrote about creating training and test data sets: https://blog.griddynamics.com/creating-training-and-test-data-sets-and-preparing-the-data-for-twitter-stream-sentiment-analysis-of-social-movie-reviews

$\endgroup$
0
$\begingroup$

The Stanford Sentiment Analysis dataset is based on Rotten Tomatoes reviews, has parses and sentiment annotation down to the syntactic component level.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.