I have attached a figure that contains 6 subplots below. Each shows training and test loss over multiple epochs. Just by looking at each graph, how can I see which one is the best? Which ones are overfitting or underfitting. Which ones are getting worse with each epoch?
Assuming that the train and validation sets in the curves under comparison are the same, the best curve is probably the one with the lowest validation loss value.
Numbering your figures from left to right and from top to bottom, I would say the best one is #5 (second row, second column).
Now, let's break down what is going on in each plot:
Very high values, seemingly random, no decrease whatsoever in either train or validation losses: the model is not learning; probably there's something wrong with either the model or the optimization process, or maybe some hyperparameter value is terribly wrong.
Descending values for both training and validation losses, with validation loss having a gap with the training one, and both stabilized (i.e. neither of them will probably go any lower -if in doubt about this, leave them more training time-): training seems Ok, but there is room for improvement if you regularize your model so that you get your training curve upper and the validation one lower.
Initially, both curves descend, then validation starts going up around step 800: overfitting. You should try regularizing the model and, if that's not effective, use early stopping to use the model that performs best on the validation data. You may also try with some hyperparameter tunning, or having a learning rate schedule that makes it smaller with time.
Both curves are descending and it seems they will keep doing that for a while: training is not finished, leave it training more time.
Both curves descend, despite the initial plateau, and reach a low point, with no gap between training and validation curves: you can probably improve the model weight initialization. Anyway, this plot seems the best, as the validation curve reaches the lowest value and there is no overfitting.
Both curves go up: there is something wrong, probably in how you define your loss function optimization process.
I don't see any clear case of underfitting among your plots. In an underfitting scenario, we would see that the model learns something but both the training and validation losses stabilize at too high values. This would suggest a lack of capacity of the model, preventing it to properly capture the data distribution with respect to the labels.
The optimal graph is the one where the graphs of train and cv losses are on top of each other. In this case, you can be sure that they are not overfitting because the model is performing as good as it did on the training set. Hence the loss curves sits on top of each other. But they can very well be underfitting.
One simple way to understand overfit and underfit is:
1) If your train error decreases, while your cv error increases, You are overfitting
2) If train and cv error both increase, You are underfitting
Coming in the clockwise direction of the graphs you posted, the first(left topmost) image is understood as a non overfit model as the graphs sit on top of each other but since the graph has been in other words "zoomed in" in a particular region, We are seeing a jittery image and hence we cannot come to a clear conclusion. The second graph is looking fine as the gap is closing in and the third image is starting to overfit as the gap between the graph is starting to widen. The next graph(down rightmost) has an increasing loss which is bad, When the train and cv loss is increasing, We can say that the model is underfitting. The next graph with a slow start seems to have converged very well. Last graph I would say has very slow convergence or there is problem with convergence for which we need to look through the model and the data.