2
$\begingroup$

I have attached a figure that contains 6 subplots below. Each shows training and test loss over multiple epochs. Just by looking at each graph, how can I see which one is the best? Which ones are overfitting or underfitting. Which ones are getting worse with each epoch?

Following is the image that contains 6 graph plots.Eac

$\endgroup$
1
$\begingroup$

Assuming that the train and validation sets in the curves under comparison are the same, the best curve is probably the one with the lowest validation loss value.

Numbering your figures from left to right and from top to bottom, I would say the best one is #5 (second row, second column).

Now, let's break down what is going on in each plot:

  1. Very high values, seemingly random, no decrease whatsoever in either train or validation losses: the model is not learning; probably there's something wrong with either the model or the optimization process, or maybe some hyperparameter value is terribly wrong.

  2. Descending values for both training and validation losses, with validation loss having a gap with the training one, and both stabilized (i.e. neither of them will probably go any lower -if in doubt about this, leave them more training time-): training seems Ok, but there is room for improvement if you regularize your model so that you get your training curve upper and the validation one lower.

  3. Initially, both curves descend, then validation starts going up around step 800: overfitting. You should try regularizing the model and, if that's not effective, use early stopping to use the model that performs best on the validation data. You may also try with some hyperparameter tunning, or having a learning rate schedule that makes it smaller with time.

  4. Both curves are descending and it seems they will keep doing that for a while: training is not finished, leave it training more time.

  5. Both curves descend, despite the initial plateau, and reach a low point, with no gap between training and validation curves: you can probably improve the model weight initialization. Anyway, this plot seems the best, as the validation curve reaches the lowest value and there is no overfitting.

  6. Both curves go up: there is something wrong, probably in how you define your loss function optimization process.

I don't see any clear case of underfitting among your plots. In an underfitting scenario, we would see that the model learns something but both the training and validation losses stabilize at too high values. This would suggest a lack of capacity of the model, preventing it to properly capture the data distribution with respect to the labels.

$\endgroup$
0
$\begingroup$

The optimal graph is the one where the graphs of train and cv losses are on top of each other. In this case, you can be sure that they are not overfitting because the model is performing as good as it did on the training set. Hence the loss curves sits on top of each other. But they can very well be underfitting.

One simple way to understand overfit and underfit is:

1) If your train error decreases, while your cv error increases, You are overfitting

2) If train and cv error both increase, You are underfitting

Coming in the clockwise direction of the graphs you posted, the first(left topmost) image is understood as a non overfit model as the graphs sit on top of each other but since the graph has been in other words "zoomed in" in a particular region, We are seeing a jittery image and hence we cannot come to a clear conclusion. The second graph is looking fine as the gap is closing in and the third image is starting to overfit as the gap between the graph is starting to widen. The next graph(down rightmost) has an increasing loss which is bad, When the train and cv loss is increasing, We can say that the model is underfitting. The next graph with a slow start seems to have converged very well. Last graph I would say has very slow convergence or there is problem with convergence for which we need to look through the model and the data.

$\endgroup$
5
  • $\begingroup$ It is very important to note that in your first paragraph you're 50% right, and it can lead to missleading concepts, which are very important. It is true that if the val loss and the train loss are close, there are no overfitting, but there can be underfitting. The underfitting case appear when a model is performing bad with respect to a desired metric, and it can be an equally bad model in both training and validation datasets $\endgroup$ – ignatius May 16 '19 at 7:56
  • $\begingroup$ Thanks! I forgot that. I will make changes to my answer so as not to mislead. $\endgroup$ – karthikeyan mg May 16 '19 at 8:12
  • $\begingroup$ You're welcome, the down vote will disappear as long as the answer is edited :) $\endgroup$ – ignatius May 16 '19 at 8:22
  • $\begingroup$ Made the changes! I request you to check it and suggest me if anything needs to edited again. :) $\endgroup$ – karthikeyan mg May 16 '19 at 8:24
  • $\begingroup$ the first graph(upper left most) seems to be having a very large learning rate or momentum. That's why learning is fluctuating greatly. $\endgroup$ – Muhammad Hanif Sarwari May 16 '19 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.