How to know the model has started overfitting?

Question

I hope the following excerpts will provide an insight into what my question is going to be. These are from here.

The learning then gradually slows down. Finally, at around epoch 280 the classification accuracy pretty much stops improving. Later epochs merely see small stochastic fluctuations near the value of the accuracy at epoch 280. Contrast this with the earlier graph, where the cost associated to the training data continues to smoothly drop. If we just look at that cost, it appears that our model is still getting "better". But the test accuracy results show the improvement is an illusion. Just like the model that Fermi disliked, what our network learns after epoch 280 no longer generalizes to the test data. And so it's not useful learning. We say the network is overfitting or overtraining beyond epoch 280.

We are training a neural network and the cost (on training data) is dropping till epoch 400 but the classification accuracy is becoming static (barring a few stochastic fluctuations) after epoch 280 so we conclude that model is overfitting on training data post epoch 280.

We can see that the cost on the test data improves until around epoch 15, but after that it actually starts to get worse, even though the cost on the training data is continuing to get better. This is another sign that our model is overfitting. It poses a puzzle, though, which is whether we should regard epoch 15 or epoch 280 as the point at which overfitting is coming to dominate learning? From a practical point of view, what we really care about is improving classification accuracy on the test data, while the cost on the test data is no more than a proxy for classification accuracy. And so it makes most sense to regard epoch 280 as the point beyond which overfitting is dominating learning in our neural network.

As opposed to classification accuracy on test data compared with training cost previously we are now placing cost on test data against training cost.

Then the book goes on to explain why 280 is the right epoch where the overfitting has started. That is what I have an issue with. I can't wrap my head around this.

We are asking model to minimize the cost and thus cost is the metric it uses as a measure of its own strength to classify correctly. If we consider 280 as the right epoch where the overfitting has started, have we not in a way created a biased model that though is a better classifier on the particular test data but nonetheless is making decisions with low confidence and hence is more prone to deviate from the results shown on the test data?

Answer 1

Let's say we want to predict if a student will land a job interview based on her resume.

Now, assume we train a model from a dataset of 10,000 resumes and their outcomes.

Next, we try the model out on the original dataset, and it predicts outcomes with 99% accuracy… wow!

But now comes the bad news.

When we run the model on a new (“unseen”) dataset of resumes, we only get 50% accuracy… uh-oh!

Our model doesn’t generalize well from our training data to unseen data.

This is known as overfitting, and it’s a common problem in machine learning and data science.

Overfitting V/s Underfitting

We can understand overfitting better by looking at the opposite problem, underfitting.

Underfitting occurs when a model is too simple – informed by too few features or regularized too much – which makes it inflexible in learning from the dataset.

Simple learners tend to have less variance in their predictions but more bias towards wrong outcomes (see: The Bias-Variance Tradeoff).

On the other hand, complex learners tend to have more variance in their predictions.

Both bias and variance are forms of prediction error in machine learning.

Typically, we can reduce error from bias but might increase error from variance as a result, or vice versa.

This trade-off between too simple (high bias) vs. too complex (high variance) is a key concept in statistics and machine learning, and one that affects all supervised learning algorithms.

Answer 2

Something that I have learned the hard way is to plot the learning curves, I know, it is not as fun as writing the machine learning code per-se, but it is fundamental to visually understand what is happening.

A rule of thumb definition is that over fitting occurs when your train accuracy keeps improving while your validation accuracy stops improving (or even starts getting worse).

Simplest solution to avoid over fitting is early stopping (stop training as soon as things look bad), of course being the simplest solution comes at a cost: it is not the best solution. Regularisation and dropout are good tools to fight over fitting, but that is a different matter :)

Hope it helps

Answer 3

As the source you are quoting says "the cost on the test data is no more than a proxy for classification accuracy." You might ask, why should we use a proxy, why not use the accuracy directly? The answer is that you need to minimize the cost function with respect to the weights and biases. Therefore it has to be a differentiable function of the the weights and biases. Accuracy is not a differentiable function and therefore cannot be used directly. But since ultimately you care about accuracy, as you yourself illustrated above (...please note the classification accuracy is 100% in the first instance and yet the cost is higher... ), you determine overfitting based on the accuracy on the test set.

Answer 4

To understand what overfitting means and how it affects the accuracy of the model, you need to understand the bias - variance tradeoff.

Under-fitting as well as overfitting are two different problems that are directly related to biased- variance problems. It is always important to understand the relation between three different factors and how these factors are connected to bias-variance ( overfitting- under-fitting) problems:

1- the size of the model. Number of parameters

2- the amount of data available for training. Number of training samples.

3- the number of iterations. training iterations.

Making a direct connection between any of these factors to overfitting- under-fitting problems without looking at the others will always lead to wrong conclusions.

Because of understanding these factors and linking theme by using mathematical equations to avoid overfitting and under-fitting problems is a difficult task, more over it is a task dependent, people use simple methods to discover and avoid overfitting. The easy way is to divide the data into three different parts, training, validation and testing. Testing should not be touched. Use training set to train the network and validation set to test the network after each iteration or a number of iterations. Theoretically, you will see that the error on the validation set decreases gradually for the first N iterations and then will be stable for very few iterations and then starts increasing. When the error starts increasing, your network starts overfitting the training data and the training process should be stopped.

Note: the value N is very related to the three factors I listed above. It is always a good practice to have a demo training set and test with different models, training data. You will see that the larger the model - the less training data the smaller the N. The smaller the model - the more training data the larger the N. Note: be careful when using small models of having under-fitting problem.