4
$\begingroup$

I have a graph which plots training datasize on X axis and accuracy on y axis. I plotted the curves using sklearn's learning_curve.

enter image description here

It is observed that the accuracy of training dataset decreases but the accuracy of validation dataset increases. I am not able to justify this behavior.

  1. Normally, as training dataset increases, the training accuracy is supposed to increase right?
  2. Also, assuming that the dataset is very noisy and hence training accuracy is decreasing as dataset size increases. But this doesn't explain why validation accuracy increases because noise is supposed to affect that too.
$\endgroup$
3
$\begingroup$

I think what you're seeing is normal behaviour:

With only few samples (like 2000) it's easy for a model to (over)fit the data - but it doesn't generalize well. So you get high training accuracy, but the model might not work well with new data (i.e. low validation/test accuracy).

As you add more samples (like 9000) it becomes harder for the model to fit the data - so you get a lower training accuracy, but the model will work better with new data (i.e. validation/test accuracy starts to rise).

So:

  1. As the training dataset increases, the training accuracy is supposed to decrease because more data is harder to fit well.

  2. As the training dataset increases, the validation/test accuracy is supposed to increase as well since less overfitting means better generalization.

Andrew Ng has a video about learning curves. Note that he plots the error on the y axis, you have the accuracy on the y axis.. so the y axis is flipped.

Also take a look at the second half of the video. It explains high bias and high variance problems.

Your model seems to have high variance (due to the big "gap" between the two curves) - it's still too complex for the small amount of data you've got. Either getting more data or using a simpler model (or more regularization on the same model) might improve the results.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.