I have a dataset of ~100k news articles, and I'm trying to build a classifier based on the headlines and leads of every article. The dataset is not pre-labeled so I'm manually labelling a subset of the articles to train on. So far, I have 9 categories with ~600 labeled articles for each category. I'm using doc2vec to create document vectors and the SVC classifier from sklearn to make predictions.

My cross validation score (with 10 splits and shuffling of data) is hovering around 0.89.

When I plot a learning curve for the training set and test set, I interpret it as the classifier suffering from high variance, and that I need to collect more data. But is there any way I can approximate how much more data is needed to get a cross validation score of say, 0.95?

Here is my learning curve and the mean scores:

enter image description here

Training scores: 1.0, 0.99919679, 0.9991984 , 0.99791667, 0.99615385, 0.99550155, 0.99487179.

Testing scores: 0.40769912, 0.72283119, 0.78529511, 0.83461723, 0.85527486, 0.86151579, 0.86471912.

EDIT: I performed a grid search on the C and gamma parameters to the SVC model, and adjusted gamma to 0.015 which made the two lines converge a bit more. Adding plot and new scores.

Learning curve after adjusting gamme

New training scores: 0.97758621, 0.98703072, 0.98177172, 0.97540872, 0.96717739, 0.96479244, 0.96241702.

New testing scores: 0.45800254, 0.78917394, 0.83526468, 0.853326, 0.86849859, 0.87232875, 0.87508134.

  • $\begingroup$ Now you have high variance problem, you have to employ generalization techniques, if you are using SVM, you can change the lambda hyper-parameter. You have to tune it to find a model to generalize well. I guess your data is enough based on the reported accuracy. $\endgroup$ – Media May 1 '18 at 18:47
  • $\begingroup$ What is the horizontal axis? $\endgroup$ – Media May 1 '18 at 18:50
  • $\begingroup$ Can you label the red and green lines? $\endgroup$ – Brian Spiering May 1 '18 at 19:01
  • $\begingroup$ I've added axes labels and a legend. $\endgroup$ – miroli May 1 '18 at 21:50

I think you already have enough data. Your problem seems to be one of generalization, which is in general a tricky one to solve. For SVM, depending on what kind you have, you could try tinkering with the parameters and kernels. Have a look here. -- https://stats.stackexchange.com/questions/35276/svm-overfitting-curse-of-dimensionality?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.