Regarding Training loss and validation loss

Question

I am trying to use keras for multi label news classification. I am a beginner in machine learning so please bear with me.

Here is my training loss vs validation loss diagram:

I understand that my model is overfitting. On the test set, I have precision of 0.89 , recall of 0.82, f1score 0.83. This according to me is high precision and high recall. So this means a good classification right ?

Further, this is the parameter settings:

My question is why in the test set there is high precision and high recall but the model is overfitting in validation set. Isn't it odd? or am i missing something?

I would have uploaded my dataset. But the dataset excel file is 800 MB.

Note that I don't have a seperate training set, validation set and test set. I have a single file. On the first column are the articles and in the second columns are the labels.

I used sklearn.train_test_split twice so that the training set is 60% test set is 20% and validation set is 20%.

Answer 1

To the learning-curves look exactly like what you would expect. The training-loss goes down to zero. That means your model is sufficient to fit the data. If the training-loss would get stuck somewhere, that would mean the model is not able to fit the data. So, your model is flexible enough. Furthermore the validation-loss goes down first until it reaches a minimum and than starts to rise again. If your validation set is large enough and representative you can argue that your model starts to overfit the data.

Next step could be to prevent the model to overfit. You can do that by adding some kind of regularization, dropout, batch-normalization.

Considering the test set. Instead of using two distinct values try to decide for one metric that does what you want. For example the F1 score. It is hard to say whether these values are good or not. It ultimately depends on what you define as "good enough". If 90% is sufficient for you than that is ok, but if you need a model that is 99.9% accurate it would not be good enough. Apply the metric to all sets training, validation and test. The validation error should be a good estimator of the test error unless you use it to fit your data. If you use it to identify the optimal number of training rounds it must be considered part of the training set and you would expect the error to be lower than in the test set. Again, if the test set error is bigger, usually something is off.

If your validation and test errors are very different from each other that probably means they sample different regions in the space of your data or the sets are not big enough so that stochastic effects play a role.