13
$\begingroup$

I hope the following excerpts will provide an insight into what my question is going to be. These are from here.

The learning then gradually slows down. Finally, at around epoch 280 the classification accuracy pretty much stops improving. Later epochs merely see small stochastic fluctuations near the value of the accuracy at epoch 280. Contrast this with the earlier graph, where the cost associated to the training data continues to smoothly drop. If we just look at that cost, it appears that our model is still getting "better". But the test accuracy results show the improvement is an illusion. Just like the model that Fermi disliked, what our network learns after epoch 280 no longer generalizes to the test data. And so it's not useful learning. We say the network is overfitting or overtraining beyond epoch 280.

We are training a neural network and the cost (on training data) is dropping till epoch 400 but the classification accuracy is becoming static (barring a few stochastic fluctuations) after epoch 280 so we conclude that model is overfitting on training data post epoch 280.

We can see that the cost on the test data improves until around epoch 15, but after that it actually starts to get worse, even though the cost on the training data is continuing to get better. This is another sign that our model is overfitting. It poses a puzzle, though, which is whether we should regard epoch 15 or epoch 280 as the point at which overfitting is coming to dominate learning? From a practical point of view, what we really care about is improving classification accuracy on the test data, while the cost on the test data is no more than a proxy for classification accuracy. And so it makes most sense to regard epoch 280 as the point beyond which overfitting is dominating learning in our neural network.

As opposed to classification accuracy on test data compared with training cost previously we are now placing cost on test data against training cost.

Then the book goes on to explain why 280 is the right epoch where the overfitting has started. That is what I have an issue with. I can't wrap my head around this.

We are asking model to minimize the cost and thus cost is the metric it uses as a measure of its own strength to classify correctly. If we consider 280 as the right epoch where the overfitting has started, have we not in a way created a biased model that though is a better classifier on the particular test data but nonetheless is making decisions with low confidence and hence is more prone to deviate from the results shown on the test data?

$\endgroup$
  • $\begingroup$ The model is not aware of the test set. It stands in as a proxy for unseen data. Therefore, if it comes from a representative distribution, you can use it to determine when overfitting occurs. If you wish, you can create yet another hold out set and see if this assumption holds. $\endgroup$ – Emre May 22 '17 at 23:27
  • $\begingroup$ What do you mean by 'making decisions with low confidence'? $\endgroup$ – Grasshopper May 24 '17 at 14:05
  • $\begingroup$ @Grasshopper let us say the model is trying to predict one of 4 classes {A, B, C, D}. Test data labels (in order) are (A, B, C, D). Now in one instance the model throws probabilities as (I will be labeling the predictions along) ((0.28, 0.24, 0.24, 0.24)(A), (0.24,0.28,0.24,0.24)(B), (0.24,0.24,0.28,0.24)(C), (0.24,0.24,0.24,0.28)(D)) and in another the model throws ((1,0,0,0)(A), (0,1,0,0)(B), (0.24,0.26,0.25,0.25)(B), (0,0,0,1)(D)). What i mean by low confidence is the first instance. please note the classification accuracy is 100% in the first instance and yet the cost is higher $\endgroup$ – Nitin Siwach May 24 '17 at 14:54
  • $\begingroup$ @Grasshopper In a nutshell. The first instance of the model is created after 280 epoch (refer to the question asked) and the second instance of the model is created after 15 epoch. Now the book goes on to suggest epoch 280 as the one where the over-fitting has started. I am finding it hard to swallow that. any help or thoughts that you can provide are much appreciated. $\endgroup$ – Nitin Siwach May 24 '17 at 15:01
6
$\begingroup$

Let's say we want to predict if a student will land a job interview based on her resume.

Now, assume we train a model from a dataset of 10,000 resumes and their outcomes.

Next, we try the model out on the original dataset, and it predicts outcomes with 99% accuracy… wow!

But now comes the bad news.

When we run the model on a new (“unseen”) dataset of resumes, we only get 50% accuracy… uh-oh!

Our model doesn’t generalize well from our training data to unseen data.

This is known as overfitting, and it’s a common problem in machine learning and data science.

Overfitting V/s Underfitting

We can understand overfitting better by looking at the opposite problem, underfitting.

Underfitting occurs when a model is too simple – informed by too few features or regularized too much – which makes it inflexible in learning from the dataset.

Simple learners tend to have less variance in their predictions but more bias towards wrong outcomes (see: The Bias-Variance Tradeoff).

On the other hand, complex learners tend to have more variance in their predictions.

Both bias and variance are forms of prediction error in machine learning.

Typically, we can reduce error from bias but might increase error from variance as a result, or vice versa.

This trade-off between too simple (high bias) vs. too complex (high variance) is a key concept in statistics and machine learning, and one that affects all supervised learning algorithms.

| improve this answer | |
$\endgroup$
2
$\begingroup$

Something that I have learned the hard way is to plot the learning curves, I know, it is not as fun as writing the machine learning code per-se, but it is fundamental to visually understand what is happening.

A rule of thumb definition is that over fitting occurs when your train accuracy keeps improving while your validation accuracy stops improving (or even starts getting worse).

Simplest solution to avoid over fitting is early stopping (stop training as soon as things look bad), of course being the simplest solution comes at a cost: it is not the best solution. Regularisation and dropout are good tools to fight over fitting, but that is a different matter :)

Hope it helps

| improve this answer | |
$\endgroup$
2
$\begingroup$

As the source you are quoting says "the cost on the test data is no more than a proxy for classification accuracy." You might ask, why should we use a proxy, why not use the accuracy directly? The answer is that you need to minimize the cost function with respect to the weights and biases. Therefore it has to be a differentiable function of the the weights and biases. Accuracy is not a differentiable function and therefore cannot be used directly. But since ultimately you care about accuracy, as you yourself illustrated above (...please note the classification accuracy is 100% in the first instance and yet the cost is higher... ), you determine overfitting based on the accuracy on the test set.

| improve this answer | |
$\endgroup$
0
$\begingroup$

To understand what overfitting means and how it affects the accuracy of the model, you need to understand the bias - variance tradeoff.

Under-fitting as well as overfitting are two different problems that are directly related to biased- variance problems. It is always important to understand the relation between three different factors and how these factors are connected to bias-variance ( overfitting- under-fitting) problems:

1- the size of the model. Number of parameters

2- the amount of data available for training. Number of training samples.

3- the number of iterations. training iterations.

Making a direct connection between any of these factors to overfitting- under-fitting problems without looking at the others will always lead to wrong conclusions.

Because of understanding these factors and linking theme by using mathematical equations to avoid overfitting and under-fitting problems is a difficult task, more over it is a task dependent, people use simple methods to discover and avoid overfitting. The easy way is to divide the data into three different parts, training, validation and testing. Testing should not be touched. Use training set to train the network and validation set to test the network after each iteration or a number of iterations. Theoretically, you will see that the error on the validation set decreases gradually for the first N iterations and then will be stable for very few iterations and then starts increasing. When the error starts increasing, your network starts overfitting the training data and the training process should be stopped.

Note: the value N is very related to the three factors I listed above. It is always a good practice to have a demo training set and test with different models, training data. You will see that the larger the model - the less training data the smaller the N. The smaller the model - the more training data the larger the N. Note: be careful when using small models of having under-fitting problem.

| improve this answer | |
$\endgroup$
  • $\begingroup$ You have said "Theoretically, you will see that the error on the validation set decreases gradually for the first N iterations and then will be stable for very few iterations and then starts increasing." What do you mean by error here. That exactly is the issue i have raised in the question. The answer is 15 if i take cost as the measure of the error and 280 if i take classification accuracy as the measure of the error $\endgroup$ – Nitin Siwach May 24 '17 at 18:12
  • $\begingroup$ please note correction in the above comment: The answer to when the overfitting has started is epoch 15 if i take cost as the measure of the error and epoch 280 if i take classification accuracy as the measure of the error $\endgroup$ – Nitin Siwach May 24 '17 at 19:11
  • $\begingroup$ The error on the validation set $\endgroup$ – Bashar Haddad May 25 '17 at 1:37
  • $\begingroup$ and the measure of that error is? (100 - Classification accuracy) or cost. I understand the error on the validation set means in connection to classification accuracy. But that exactly is what i am asking. Why not the cost? Please refer to the comments i have made in response to Grasshopper on the original question $\endgroup$ – Nitin Siwach May 25 '17 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.