1
$\begingroup$

How can I check if a bigger training data set would improve my accuracy of my scikit classifier, is there a method or something?

$\endgroup$
1
  • $\begingroup$ Do you mean actual accuracy or model performance in general? $\endgroup$ – Roman Jan 16 '19 at 14:01
3
$\begingroup$

One idea:

  1. Split your data into train / hold out datasets.
  2. Train the model on a fraction of the training data (say 50%) and test on the holdout dataset.
  3. Train the model on a larger fraction of the training data (say 75%) and test on the holdout dataset.

It's important that you use the same holdout data for testing so you can perform a true test of accuracy.

Since you're doing classification, you should check that your data is balanced, and adjust if not (this may also improve your accuracy without needing larger training data).

$\endgroup$
3
$\begingroup$

The Validation Curve method (available on Scikit) plots the cross-validation score of your metric as you increase the number of training examples. If the model performance starts stagnating with the training examples of your original dataset, it may be a symptom that a bigger dataset will not improve your classifier's performance.

This also allows you to clearly observe the Bias vs Variance behaviour of your model.

Validation Curve

As shown in the image below (source), you have a high bias (underfitting) when the both training and validation performances are clearly below your target. On the other side, you can overfit and cause your model to perform much better on the training dataset than in the validation, causing high variance (aka overfitting).

A well trained model will perform with a good Bias vs Variance trade-off, both performing near the desired target and performing evenly in both training and validation datasets.

enter image description here

$\endgroup$
4
  • $\begingroup$ Can I plot such a learning curves diagramm for the scikit MLPCassifier too? $\endgroup$ – jochen6677 Feb 1 '19 at 10:12
  • 2
    $\begingroup$ Yes, the Learning Curve is agnostic to the model. Check this example here $\endgroup$ – UrbanoFonseca Feb 1 '19 at 10:23
  • $\begingroup$ But how shall I interpret the diagramm above: Does it tell me that if I would collect about 1200 training samples in total this would be the optimum number of training samples because further training samples would not improove my accuracy? $\endgroup$ – jochen6677 Feb 4 '19 at 10:24
  • 1
    $\begingroup$ I just updated the answer explaining the Bias vs Variance. From the 1st image, it appears that increasing the number of samples from 1200 to 1400 creates small marginal improvements to the model's performance. If you achieve your expected performance target with 1200 samples you can consider this as the most efficient training sample size. In the end you have a trade-off between number of samples (e.g. computing speed) and the model's performance (plus the original bias vs variance trade-off). $\endgroup$ – UrbanoFonseca Feb 4 '19 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.