# How much more data does my classifier need?

I have a dataset of ~100k news articles, and I'm trying to build a classifier based on the headlines and leads of every article. The dataset is not pre-labeled so I'm manually labelling a subset of the articles to train on. So far, I have 9 categories with ~600 labeled articles for each category. I'm using doc2vec to create document vectors and the SVC classifier from sklearn to make predictions.

My cross validation score (with 10 splits and shuffling of data) is hovering around 0.89.

When I plot a learning curve for the training set and test set, I interpret it as the classifier suffering from high variance, and that I need to collect more data. But is there any way I can approximate how much more data is needed to get a cross validation score of say, 0.95?

Here is my learning curve and the mean scores:

Training scores: 1.0, 0.99919679, 0.9991984 , 0.99791667, 0.99615385, 0.99550155, 0.99487179.

Testing scores: 0.40769912, 0.72283119, 0.78529511, 0.83461723, 0.85527486, 0.86151579, 0.86471912.

EDIT: I performed a grid search on the C and gamma parameters to the SVC model, and adjusted gamma to 0.015 which made the two lines converge a bit more. Adding plot and new scores.

New training scores: 0.97758621, 0.98703072, 0.98177172, 0.97540872, 0.96717739, 0.96479244, 0.96241702.

New testing scores: 0.45800254, 0.78917394, 0.83526468, 0.853326, 0.86849859, 0.87232875, 0.87508134.

• Now you have high variance problem, you have to employ generalization techniques, if you are using SVM, you can change the lambda hyper-parameter. You have to tune it to find a model to generalize well. I guess your data is enough based on the reported accuracy. May 1, 2018 at 18:47
• What is the horizontal axis? May 1, 2018 at 18:50
• Can you label the red and green lines? May 1, 2018 at 19:01
• I've added axes labels and a legend. May 1, 2018 at 21:50