test accuracy of text classification is too less

Question

I have a data set of movies and their subtitles. My task is to classify them based on their ratings - [R, NR, PG, PG-13, G].

I have 13 examples for each class. I preprocessed the subtitles in the following way:

• I used word puns tokenizer to tokenize subtitles.
• Removed stop words and punctuation.
• Performed stemming.
• Vectorized the subtitles using TF-IDF vectorizer.

The accuracy that I am getting using:

1) svm : train accuracy is 1.0 and test accuracy is .17.

2) naive bayes: training accuracy is 0.5 and test accuracy is .23.

I have the following questions:

1)Why is my accuracy so low and what can I do to improve the accuracy?

2)Will more training data help?

3)Should I perform feature selection?

4)What other classification algorithms can I use to improve the accuracy?

Answer 1

You should try to perform cross-validation using all your datasets. Maybe your test set has a completely different distribution than your training set (with such a small number of samples out of all possible subtitles, it is likely). Did you create this dataset?

What is the size of the final input vectors you are using and what size of vocabulary are you using for this?

1. At least in the case of the SVM, it seems that your model is overfitting and you need to increase the regularization.
2. More training data will always help.
3. Not sure what are your options here.
4. There are a lot of options based on the small number of samples, you can try using clustering (KNN, K-Mean).