2
$\begingroup$

I have a Keras neural network that has images both as input and reference data.

My network demonstrates overfitting (for example, train accuracy is about 80% but test accuracy is only up to 70%) due to small amount of input data relatively to network size. Nevertheless, 70% test accuracy is pretty much fine for my problem and test accuracy doesn't start decreasing after overfitting begins -- it increases along with train accuracy, but is always several percent behind it. And model simplification doesn't allow me to reach such good accuracy.

So, in my case it looks like overfitting is doing nothing bad, but I know that its observed effects may be quite different and I'd like to check if my model performs correctly.

Could you please suggest some classical ways to analyze the model quality? May be special metrics or statistical methods are usually used or some sort of entropy parameters estimation needs to be performed? Is it possible to understand whether my model learned some useful patterns from input data or just memorized the data itself?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ This happens with me a lot of times, and a generally i focus on my validation/test set accuracy. If the model is getting 85% accuracy on test set and 98% on train set then it is overfitted to some extent. But at the end validation set is the target, and validation accuracy is what I focus on. You might again test it on another test set, perhaps. $\endgroup$
    – Devashish Prasad
    May 31, 2021 at 12:32

1 Answer 1

Reset to default
0
$\begingroup$

A quite useful site is this https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/. Some other things that you could do in order to avoid overfitting is:

  1. early stoping, plot the accuracy of validation and test data sets over the time and when the validation accuracy becomes much better stop training your model.

  2. try to use L2 regularization which gives a penalty of the complexity of the model.

  3. in case that you use trees, try to prune your tree.

  4. in case of neural networks try to use dropout, https://machinelearningmastery.com/dropout-regularization-deep-learning-models-keras/.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.