I am doing text classification with Python. I have around 120 records with 2 columns:

  • text
  • class

I tokenize, stem and lematize the words, I also did some of my own text preprocessing. When I run the alghoritms using sklearn and divide it to training and test sets each time I run the script the Accuracy Score is so different each time for both alghoritms. Sometimes I get around 70%, sometimes 40%. Is it because of the number of records (120) or not necessarily? if it is about number of records how much of them should I have?


My suspicion is that you did not put random_state parameter on train_test_split. Don't worry it is quite common error, it just happen that your dataset is small and often with small dataset variability between result will be heavily amplified. Having small amount of data is OK, and sometimes we have to work with what we have, as long as you know how to handle it, just don't expect your model to generalize very well.

Your model relies on bag of words representation I presume. With small data what could go wrong is having a lot of Out of Vocabulary words (words in test-set does not available in training set), to handle this you can apply smoothing e.g. Add-one smoothing.

  • $\begingroup$ What should be the value of random_state in my case and my number of data? $\endgroup$ Nov 22 '19 at 12:45
  • $\begingroup$ random_state is random seed, it just keep your experimentation consistent for each run. There is no specific rule of how big your data should be, but I would say 1000 should be more stable. Bag of Words representation performs more stable (not necessarily better) as number of data increases. $\endgroup$ Nov 22 '19 at 12:54
  • $\begingroup$ Does sklearn have some implementation of add-one smoothing? I was thinking about creating my own set of words, some kind of dictionary, and after creating tokens stemmed etc. I would keep just words from dictionary in the dataset, is it a good idea? $\endgroup$ Nov 22 '19 at 13:10
  • $\begingroup$ No, but it is rather simple and there are many resources about this on smoothing on the internet $\endgroup$ Nov 22 '19 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.