5
$\begingroup$

I am on a binary classification problem with the AUC metrics. I did a random split 70%, 30% for training and test sets. My first attempts using random forest with default hyper-parameters gave me auc 0.85 on test set and 0.96 on training set. So, the model overfits. But the score of 0.85 is good enough for my business. I also did a 5-folds cross validation with the same model and same hyper-parameters and the test set results were consistently something between 0.84 and 0.86

My question is: can I believe on the score 0.85 and use this model in production?

$\endgroup$
3
$\begingroup$

Yes, if your 0.85 AUC is good enough for your use case this is a good enough model. The performance on the training set indicates how well your model knows the training set. This we don't really care about, it's just what the model tries to optimize. The performance on the test set is an indication on how well your model generalizes. This is what we care about, and your model gets to around 0.85 as an estimate for your generalization. Differences between training and testing are the norm and in this case it could be that you might get a better performance by adding stronger regularization but if 0.85 is good enough, go for it!

$\endgroup$
  • $\begingroup$ I would just add that if the OP can choose between two gradient boosting trees, one of which is using shorter trees than the other, I would go with the one using shorter trees. It is harder to overfit when you have less degrees of freedom. Plus, Occam's Razor. :) $\endgroup$ – Ricardo Cruz Jul 8 '16 at 9:54
2
$\begingroup$

My first attempts [...] gave me auc 0.85 on test set and 0.96 on training set. So, the model overfits.

This is not quite true.

See, (almost) each estimator will have a better prediction score on the training data than on the testing data. It doesn't mean each estimator overfit.

It is normal though to habe a better score on the training set, as the estimator is built on it, meaning its parameters are fitted thanks to it. However, your estimator can fit your training data more or less.

Let's take your Random-Forest example. If the depth is too high, you'll fit way to much to the training data : overfit. If the depth is not high enough, it will be hard to generalize to other data : you underfit.

  1. Underfitting : 0.96 on train set & 0.82 on test set
  2. Possible good fitting : 0.96 on train set & 0.89 on test set
  3. Overfitting : 0.96 on train set & 0.75 on test set

As a good data-scientist, you want your model to fit the data enough to generalize well but not too much not to overfit. To control how your model generalize, one uses cross-validation techniques. The value you get is pretty-much what you will obtain with new value ± the variance associated to this crossvalidation

PS: Using cross-validation too often on test data makes you, in a way, learning this data as you choose them to maximize your test score. It can lead to a form of overfitting for future new data.

$\endgroup$
  • $\begingroup$ I do aware about the form of overfitting you mention in your PS. In this question, I used cross-validation only to evaluate a given model, not to compare between models or to tuning any parameters. And actually it is really my first attempt. $\endgroup$ – phi Jul 4 '16 at 15:32
  • $\begingroup$ What I meant is that you cannot know if you overfit only with these results. Score on training set are systematically higher (besides uncommon distribution of data) than on testing data. It doesn't involve overfitting necessarily. $\endgroup$ – Igor OA Jul 4 '16 at 15:37
  • $\begingroup$ This is a good point. A good example would be using the median as your prediction. This model cannot overfit. It has only one degree of freedom. Yet, it will give slightly higher score for training than test in any normal, finite dataset. $\endgroup$ – Ricardo Cruz Jul 8 '16 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.