Understanding Learning Curves

Question

I would like to clarify my understanding of learning curves with two example plots below. I am experimenting with small data sets here between 500 and 1500 samples to clarify my understanding.

My understanding from the learning curve below is that underlying data must have a lot of noise and therefore the learning algorithm is not able to generalize the underlying function. This learning curve indicates neither high bias or variance. Just the overall data samples are not a good indicator of the outcome. Is my interpretation of this learning curve above correct? Would getting more data help?

From the learning curve below, I am interpreting this as a good learning curve. Because both the training and cross validation misclassification errors are going down as we add more samples. Around ~240 samples we get the least difference between the training and validation sets. Is this indicative of high variance since the training and validation never converge even though the misclassification error delta is pretty low between the two?

Answer 1

Learning curve 2 shows the typical shape of a learning curve: low training error at the beginning followed by an increase while the CV error is decreasing. Whether the absolute errors are considered low or high is always problem dependend. More specifically, it depends on two questions:

• what is achievable?
• what is the goal?

(for example: in stock market predictions the acceptable/target error would be much higher than in cat/dog image recognition)

You can still observe some overfitting due to the difference of a training error around around 0.01 and a validation error around 0.02 but I would not consider it large.

Learning curve 1 is harder to interpret but I consider it high bias*: In the beginning with a small sample size the model is supposed to achieve a very low training error. However, it fails to do so, i.e. it is not even able to learn/overfit the small training sample. To fix this I'd try to increase your model complexity or add features to your data.

---

*As Erwan wrote, you cannot distinguish between high bias and noisy features. My phrasing takes this one step further as I would not even distinguish between the two cases. High bias is high bias - which could be due to the model or the data.

Answer 2

I wouldn't interpret the first graph the same way. As far as know (and bear in mind that nobody knows everything), there's no clear way to distinguish between noisy features and bias (underfitting). But in either case there's no reason why the validation/CV loss would be much higher than the training loss, this is actually a clear sign of overfitting: if the model was not able to generalize, the training loss would be as high as the validation loss. Instead the model succeeds in representing some patterns, but it turns out that the patterns it captures don't apply (at least not as well) on the validation set. In other words, the model captures patterns which happen by chance in the training set: it overfits.

It's possible that adding more instances would help reduce the overfitting, as the CV loss appears to still decrease significantly in the 800-900 zone. But the difference is very large, so it's not sure that it can be solved only with more data.

I agree that the second graph shows some overfitting (as well), but not as much as the first one. In my opinion, the fact that the lowest difference between the two curves is not at the end of the curve points to either a model which is not stable yet (needs more data) or maybe there are simply not enough points on the curve to observe the real trend (i.e. the low difference at 240 and/or the higher one at 300 might happen by chance). This is indeed variance, but it's not very high in my opinion.