Would you consider that overfitting?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
2
No, it's not an example of overfitting! It would be overfitting if valid loss started to increase while training loss was going on to decrease.
Edit: the answer for the second question It's worth considering how auc is calculated. We have the probabilities of each instance to belong to the positive class. Then we sort these probabilities. If all positive instances appear in the first part of the sorted list and all the negative are in the second, then auc is 1 (the "perfect performance" according to auc observation).
Now let's consider loss computation. For example binary cross entropy. The formula is $loss = -1/N * \sum{y_i * log(p(y_i)) + (1 - y_i)*log(1 - p(y_i))}$ where $y_i$ - true lable, $p(y_i)$ - probability that $y_i$ belongs to the positive class. We can predict for each negative observation, that the probability is 0.998, then loss will be huge. But if predicted probabilities for positive observations are 0.999 (higher than for negative), then in terms of AUC we will have perfect performance.
That is why I guess, we have to evaluate loss.
-
$\begingroup$ Thank you Lana, though the difference in the loss between training and validation is not indicating overfitting? $\endgroup$ Aug 24, 2019 at 18:23
-
$\begingroup$ @NickolasPapanikolaou it's normal, when loss differs on training and validation set, because the model becomes more familiar with the data, which it has already seen in training set (it's the reason, why splitting into test, validation, train is usually performed). However the model performance on the test data shows, that model continues to extract some useful information, therefor the learning process is going on but slower than before. $\endgroup$– LanaAug 25, 2019 at 5:43
$\begingroup$
$\endgroup$
0
No it this isn't overfitting.
First of all the AUC is exactly the same between train and validation sets. The losses might have a gap but since the validation loss is still dropping (even if slowly) you are OK.
$\begingroup$
$\endgroup$
5
What about this one? In that case, the validation loss is increasing but AUC doesn't follow the same pattern, which one to believe loss or performance?
-
$\begingroup$ Forgot to mention the AUC on the holdout test set is 0.928 $\endgroup$ Aug 26, 2019 at 8:54
-
$\begingroup$ Which loss do you have? What is distribution of classes in train and test sets? $\endgroup$– LanaAug 26, 2019 at 12:36
-
$\begingroup$ I've answered this question, editing my first answer. In this case there's overfitting and loss performance should be evaluated $\endgroup$– LanaAug 26, 2019 at 13:10
-
$\begingroup$ So, your new question and my editions are the argument against the assumption, made by Javier: "First of all the AUC is exactly the same between train and validation sets". It couldn't be the main reason $\endgroup$– LanaAug 26, 2019 at 13:14
-
$\begingroup$ All three sets (train, validation, test) are fully balanced, I have used binary_crossentropy. I believe the problem is related to the AUC, since if I use the accuracy the curves deviate (signs of overfitting), many thanks for your answers $\endgroup$ Aug 27, 2019 at 12:21