0
$\begingroup$

I am trying to use keras for multi label news classification. I am a beginner in machine learning so please bear with me.

Here is my training loss vs validation loss diagram:

enter image description here

I understand that my model is overfitting. On the test set, I have precision of 0.89 , recall of 0.82, f1score 0.83. This according to me is high precision and high recall. So this means a good classification right ?

Further, this is the parameter settings:

enter image description here

My question is why in the test set there is high precision and high recall but the model is overfitting in validation set. Isn't it odd? or am i missing something?

I would have uploaded my dataset. But the dataset excel file is 800 MB.

Note that I don't have a seperate training set, validation set and test set. I have a single file. On the first column are the articles and in the second columns are the labels.

I used sklearn.train_test_split twice so that the training set is 60% test set is 20% and validation set is 20%.

$\endgroup$
1
$\begingroup$

To the learning-curves look exactly like what you would expect. The training-loss goes down to zero. That means your model is sufficient to fit the data. If the training-loss would get stuck somewhere, that would mean the model is not able to fit the data. So, your model is flexible enough. Furthermore the validation-loss goes down first until it reaches a minimum and than starts to rise again. If your validation set is large enough and representative you can argue that your model starts to overfit the data.

Next step could be to prevent the model to overfit. You can do that by adding some kind of regularization, dropout, batch-normalization.

Considering the test set. Instead of using two distinct values try to decide for one metric that does what you want. For example the F1 score. It is hard to say whether these values are good or not. It ultimately depends on what you define as "good enough". If 90% is sufficient for you than that is ok, but if you need a model that is 99.9% accurate it would not be good enough. Apply the metric to all sets training, validation and test. The validation error should be a good estimator of the test error unless you use it to fit your data. If you use it to identify the optimal number of training rounds it must be considered part of the training set and you would expect the error to be lower than in the test set. Again, if the test set error is bigger, usually something is off.

If your validation and test errors are very different from each other that probably means they sample different regions in the space of your data or the sets are not big enough so that stochastic effects play a role.

$\endgroup$
2
  • $\begingroup$ Thank you for the answer...what do you mean by test set error? How do you calculate it? there are about 25000 rows in validation set and test set whereas training set has about 75000 rows....This means the dataset is obviously large so it's overfitting right? $\endgroup$ – Annonymous programmer Jun 12 '19 at 17:31
  • $\begingroup$ Usually, you split you data set into three distinct and independent sets. The training set, the validation set and the test set. You use the training set to fit the model, the validation set to control overfitting and the test set for final evaluation of performance. When you run you model using the training set and compare it to the target values and you calculate the error you get the "training-error". That is how I refer to it. Similar if you run your model against the test set you and calculate the error (or whatever metric you use) you get the "test-error" $\endgroup$ – Sören Jun 12 '19 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.