7
$\begingroup$

I'm training a neural network on 'easy' dataset with ~15k examples. Network overfits pretty fast.

The thing I cannot understand that after 5th epoch validation loss is starting to worsen, while precision and recall are continue to improve for 10 more epochs. (loss = binary cross-entropy) validation loss vs precision/recall

Diving deeper:

I've checked if there is a lot of predictions around probability ~0.5, but it is not:

enter image description here

Also, there is a plot of percent of correct predictions based on prediction probability. There is some pattern here, but the number of elements is quite small to make conclusions.

enter image description here

So, my question is: why can it happen, and what to do about it?

$\endgroup$

2 Answers 2

5
$\begingroup$

What I found while digging down:

As neural network continues to train, it gets more confident in predictions. And while it overfits at certain scenarios, it makes log_loss to increase exponentially, since it was more confident while mistaken.

Meanwhile, if I look at mean average error, I would see that it continues to decrease, which indicates that network still learns something. Which at the end make f1_score to increase.

enter image description here

One important thing I noticed: it will not always hold true, because there is a tradeoff between overfitting/learning new, and MAE is weaker metric in binary outcomes.

enter image description here

$\endgroup$
0
$\begingroup$

Your precision and recall are not really moving significantly, neither is your loss. There are some predictions with probability around $0.5$, and I think that these are the reason for this slight trends.

As I see it, the quantities that you claim to be changing are almost constant, so I wouldn't worry about it. I might be wrong, though, but the changes in loss function are ridiculous.

$\endgroup$
1
  • $\begingroup$ I disagree, since my values have been also padded with zeroes, so real loss is bigger. Also, if I run this simulation multiple times, this trend continue to hold. $\endgroup$
    – Vadym B.
    Jun 5, 2018 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.