35
$\begingroup$

Let's say we have two models trained. And let's say we are looking for good accuracy. The first has an accuracy of 100% on training set and 84% on test set. Clearly over-fitted. The second has an accuracy of 83% on training set and 83% on test set.

On the one hand, model #1 is over-fitted but on the other hand it still yields better performance on an unseen test set than the good general model in #2.

Which model would you choose to use in production? The First or the Second and why?

$\endgroup$
5
  • 3
    $\begingroup$ The difference of 1% on the test set makes it easy to choose the 2nd one. But I think if the difference was 10% on the test set than people might choose the "overfit" model. $\endgroup$
    – jerlich
    Jan 13 '20 at 6:55
  • 3
    $\begingroup$ 1% difference in test accuracy will not be statistically significant for many test sets. How big is yours? $\endgroup$
    – Will
    Jan 13 '20 at 9:27
  • $\begingroup$ Can we have a third option: use the second to further improve into a third? The second is likely more salvageable than the first, but neither would be ideal in production. What is your goal? $\endgroup$
    – Mast
    Jan 13 '20 at 10:41
  • $\begingroup$ The test and train set metrics are random variables - if for example you're using k-fold cross validation then you can estimate the confidence of both using the variance for example - this could indicate that you cannot really say 1 is higher than 2 with say 90% confidence. So whilst it's a bit hand wavey I would prefer the model which generalises better $\endgroup$ Jan 13 '20 at 21:57
  • 3
    $\begingroup$ As mentioned in @Ray's answer, it is not correct that the first model is necessarily overfit. Random Forests, for example, are explicitly designed to produce this situation. $\endgroup$ Jan 17 '20 at 19:23
21
$\begingroup$

There are a couple of nuances here.

  1. Complexity question very important - ocams razor
  2. CV - is this trully the case 84%/83% (test it for train+test with CV)

Given this, personal opinion: Second one.

Better to catch general patterns. You already know that first model failed on that because of the train and test difference. 1% says nothing.

$\endgroup$
5
  • 2
    $\begingroup$ I disagree with the blanket statement that "1% says nothing". With a lot of data, that can be a statistically significant difference, and in situations with class imbalance, that might be a very important 1% that's being misclassified (although in that situation, accuracy is a terrible measure to start with). $\endgroup$ Jan 13 '20 at 14:44
  • 2
    $\begingroup$ you are reading it wrong. 1% says nothing in THIS situation. 1% in general is big difference! $\endgroup$
    – Noah Weber
    Jan 13 '20 at 14:45
  • 1
    $\begingroup$ "Better to catch general patterns. You already know that first model failed on that because of the train and test difference." No, the 84% says it caught general patterns. The 100% says those general patterns applied very well to the test set. $\endgroup$ Jan 13 '20 at 21:25
  • $\begingroup$ Could you explain what CV is in the context of your answer? $\endgroup$
    – Fritz
    Jan 20 '20 at 9:19
  • 1
    $\begingroup$ cross validation $\endgroup$
    – Noah Weber
    Jan 20 '20 at 10:05
13
$\begingroup$

It depends mostly on the problem context. If predictive performance is all you care about, and you believe the test set to be representative of future unseen data, then the first model is better. (This might be the case for, say, health predictions.)

There are a number of things that would change this decision.

  1. Interpretability / explainability. This is indirect, but parametric models tend to be less overfit, and are also generally easier to interpret or explain. If your problem lies in a regulated industry, it might be substantially easier to answer requests with a simpler model. Related, there may be some ethical concerns with high-variance models or non-intuitive non-monotonicity.

  2. Concept drift. If your test set is not expected to be representative of production data (most business uses), then it may be the case that more-overfit models suffer more quickly from model decay. If instead the test data is just bad, the test scores may not mean much in the first place.

  3. Ease of deployment. While ML model deployment options are now becoming much easier and more sophisticated, a linear model is still generally easier to deploy and monitor.

See also
Can we use a model that overfits?
What to choose: an overfit model with higher evaluation score or a non-overfit model with lower one?
https://stats.stackexchange.com/q/379589/232706
https://stats.stackexchange.com/q/220807/232706
https://stats.stackexchange.com/q/494496/232706
https://innovation.enova.com/from-traditional-to-advanced-machine-learning-algorithms/

(One last note: the first model may well be amenable to some sort of regularization, which will trade away training accuracy for a simpler model and, hopefully, a better testing accuracy.)

$\endgroup$
1
  • 5
    $\begingroup$ There's an interesting paper arxiv.org/abs/1812.11118v2 with a claim that with a sufficiently powerful network you get 'beyond overfitting' as the test accuracy starts to improve (due to regularization and other prior structural bias) after overfitting while the train accuracy stays at 100%. $\endgroup$
    – Peteris
    Jan 13 '20 at 14:18
8
$\begingroup$

The first has an accuracy of 100% on training set and 84% on test set. Clearly over-fitted.

Maybe not. It's true that 100% training accuracy is usually a strong indicator of overfitting, but it's also true that an overfit model should perform worse on the test set than a model that isn't overfit. So if you're seeing these numbers, something unusual is going on.

If both model #1 and model #2 used the same method for the same amount of time, then I would be rather reticent to trust model #1. (And if the difference in test error is only 1%, it wouldn't be worth the risk in any case; 1% is noise.) But different methods have different characteristics with regard to overfitting. When using AdaBoost, for example, test error has often been observed not only to not increase, but actually continue decreasing even after the training error has gone to 0 (An explanation of which can be found in Schapire et. al. 1997). So if model #1 used boosting, I would be much less worried about overfitting, whereas if it used linear regression, I'd be extremely worried.

The solution in practice would be to not make the decision based only on those numbers. Instead, retrain on a different training/test split and see if you get similar results (time permitting). If you see approximately 100%/83% training/test accuracy consistently across several different training/test splits, you can probably trust that model. If you get 100%/83% one time, 100%/52% the next, and 100%/90% a third time, you obviously shouldn't trust the model's ability to generalize. You might also keep training for a few more epochs and see what happens to the test error. If it is overfitting, the test error will probably (but not necessarily) continue increasing.

$\endgroup$
1
  • 1
    $\begingroup$ This event isn't unusual. The 100% accuracy is, but the pattern isn't. In general, as you approach the minimum generalization error for a model, the gap between train and test performance widens. The absolute value of that gap will vary by model architecture, as you mentioned in your post, but the pattern is the same. $\endgroup$ Mar 12 at 1:46
3
$\begingroup$

Obviously the answer is highly subjective; in my case clearly the SECOND. Why? There's nothing worse than seeing a customer running a model in production and not performing as expected. I've had literally had a technical CEO who wanted to get a report of how many customers have left in a given month and the customer churn prediction model. It was not fun :-(. Since then, I strongly favor high bias/low variance models.

$\endgroup$
3
$\begingroup$

These numbers suggest that the first model is not, in fact, overfit. Rather, it suggests that your training data had few data points near the decision boundary. Suppose you're trying to classify everyone as older or younger than 13 y.o. If your test set contains only infants and sumo wrestlers, then "older if weight > 100 kg, otherwise younger" is going to work really well on the test set, not so well on the general population.

The bad part of overfitting isn't that it's doing really well on the test set, it's that it's doing poorly in the real world. Doing really well on the test set is an indicator of this possibility, not a bad thing in and of itself.

If I absolutely had to choose one, I would take the first, but with trepidation. I'd really want to do more investigation. What are the differences between train and test set, that are resulting in such discrepancies? The two models are both wrong on about 16% of the cases. Are they the same 16% of cases, or are they different? If different, are there any patterns about where the models disagree? Is there a meta-model that can predict better than chance which one is right when they disagree?

$\endgroup$
1
  • $\begingroup$ I think you mean to say train instead of test in several parts of this answer. $\endgroup$ Mar 12 at 2:30
1
$\begingroup$

It seems a lot of people misunderstand overfitting here. Overfitting is not the gap between train and test performance. Overfitting is when you add complexity to a model, and there is no return on investment, or, most times, a loss in return.

See page 38 of Elements of Statistical Learning for the graph below. An overfit model is one that is to the right of the minimum of test error. Notice that for the best-fitting model, the gap between train and test is still relatively high. The correct answer is to choose the model with 84% accuracy, assuming you know that the 1% difference is statistically significant. I'm of course also assuming interpretability is not of concern here.

enter image description here

$\endgroup$
0
$\begingroup$

If your options are indeed "100% on train / 84% on validation" vs "83% on train / 83% on validation", I'd feel safer with the second one - but really, I'd take a third option: Try and tweak the first model to reduce overfitting (with the usual methods), hopefully squeezing a bit more accuracy out of it.

$\endgroup$
0
$\begingroup$

Primarily, go for CV for the training and test set. If you still get the same type of result, then choose the second model.

The first model has a very large difference in accuracy between the training and test set. It is a very specific model. There is a chance that the high accuracy on the test set appeared due to data leakage.

The second model is a more general purpose model with acceptable accuracy results on both sets.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.