8
$\begingroup$

Would you consider that overfitting?

enter image description here

$\endgroup$
0
$\begingroup$

No, it's not an example of overfitting! It would be overfitting if valid loss started to increase while training loss was going on to decrease.

Edit: the answer for the second question It's worth considering how auc is calculated. We have the probabilities of each instance to belong to the positive class. Then we sort these probabilities. If all positive instances appear in the first part of the sorted list and all the negative are in the second, then auc is 1 (the "perfect performance" according to auc observation).

Now let's consider loss computation. For example binary cross entropy. The formula is $loss = -1/N * \sum{y_i * log(p(y_i)) + (1 - y_i)*log(1 - p(y_i))}$ where $y_i$ - true lable, $p(y_i)$ - probability that $y_i$ belongs to the positive class. We can predict for each negative observation, that the probability is 0.998, then loss will be huge. But if predicted probabilities for positive observations are 0.999 (higher than for negative), then in terms of AUC we will have perfect performance.

That is why I guess, we have to evaluate loss.

$\endgroup$
  • $\begingroup$ Thank you Lana, though the difference in the loss between training and validation is not indicating overfitting? $\endgroup$ – Nickolas Papanikolaou Aug 24 '19 at 18:23
  • $\begingroup$ @NickolasPapanikolaou it's normal, when loss differs on training and validation set, because the model becomes more familiar with the data, which it has already seen in training set (it's the reason, why splitting into test, validation, train is usually performed). However the model performance on the test data shows, that model continues to extract some useful information, therefor the learning process is going on but slower than before. $\endgroup$ – Lana Aug 25 '19 at 5:43
4
$\begingroup$

No it this isn't overfitting.

First of all the AUC is exactly the same between train and validation sets. The losses might have a gap but since the validation loss is still dropping (even if slowly) you are OK.

$\endgroup$
1
$\begingroup$

What about this one? In that case, the validation loss is increasing but AUC doesn't follow the same pattern, which one to believe loss or performance?

enter image description here

$\endgroup$
  • $\begingroup$ Forgot to mention the AUC on the holdout test set is 0.928 $\endgroup$ – Nickolas Papanikolaou Aug 26 '19 at 8:54
  • $\begingroup$ Which loss do you have? What is distribution of classes in train and test sets? $\endgroup$ – Lana Aug 26 '19 at 12:36
  • $\begingroup$ I've answered this question, editing my first answer. In this case there's overfitting and loss performance should be evaluated $\endgroup$ – Lana Aug 26 '19 at 13:10
  • $\begingroup$ So, your new question and my editions are the argument against the assumption, made by Javier: "First of all the AUC is exactly the same between train and validation sets". It couldn't be the main reason $\endgroup$ – Lana Aug 26 '19 at 13:14
  • $\begingroup$ All three sets (train, validation, test) are fully balanced, I have used binary_crossentropy. I believe the problem is related to the AUC, since if I use the accuracy the curves deviate (signs of overfitting), many thanks for your answers $\endgroup$ – Nickolas Papanikolaou Aug 27 '19 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.