Why validation loss worsens while precision/recall continue to improve?

Question

I'm training a neural network on 'easy' dataset with ~15k examples. Network overfits pretty fast.

The thing I cannot understand that after 5th epoch validation loss is starting to worsen, while precision and recall are continue to improve for 10 more epochs. (loss = binary cross-entropy)

Diving deeper:

I've checked if there is a lot of predictions around probability ~0.5, but it is not:

Also, there is a plot of percent of correct predictions based on prediction probability. There is some pattern here, but the number of elements is quite small to make conclusions.

So, my question is: why can it happen, and what to do about it?

Answer 1

What I found while digging down:

As neural network continues to train, it gets more confident in predictions. And while it overfits at certain scenarios, it makes log_loss to increase exponentially, since it was more confident while mistaken.

Meanwhile, if I look at mean average error, I would see that it continues to decrease, which indicates that network still learns something. Which at the end make f1_score to increase.

One important thing I noticed: it will not always hold true, because there is a tradeoff between overfitting/learning new, and MAE is weaker metric in binary outcomes.

Answer 2

Your precision and recall are not really moving significantly, neither is your loss. There are some predictions with probability around $0.5$, and I think that these are the reason for this slight trends.

As I see it, the quantities that you claim to be changing are almost constant, so I wouldn't worry about it. I might be wrong, though, but the changes in loss function are ridiculous.