0
$\begingroup$

I am confused what I should take into account while trying to detect overfitting of a model.

Let's say I have a classification problem with the main metric being ROC-AUC. I split the data into train and test sets. I perform cross-validation on the training set and collect the average metric and the model with the best parameters. Then I use this model to predict X_test.

CV metric: ~0.75 ROC-AUC

Test: ~0.74 ROC-AUC

But when I do model(best_parameters).fit(X_train, y_train) and then .predict_proba(X_train), I get ROC-AUC = 1.0. Also, during cross-validation, the train-folds metric is 1.0.

Does it mean the model is overfitted if my training metrics = 1.0? Or I should not judging by train metrics at all? Should also monitor loss function?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

But when I do model(best_parameters).fit(X_train, y_train) and then .predict_proba(X_train), I get ROC-AUC = 1.0.

Yes, this would mean that the model is fitting your training data perfectly. That is overfitting. The model can fit the training data perfectly, and the predictions are exactly the input of the data.

Also, during cross-validation, the train-folds metric is 1.0.

This is strange. When cross validating, the model is not trained and evaluated on the same data. So while the results of cross-validation may be a little bit higher than your final testing results, the difference is too high. But are you sure this is what you mean, I understand this to be contradicting

CV metric: ~0.75 ROC-AUC

Test: ~0.74 ROC-AUC

0.75 would make sense for the cross validation. That means that the model is very flexible, so it is can overfit on the training data. So conventional wisdom is to try to reduce overfitting by making it less flexible. This should bring the training error closer to the testing error. But, also see Can we use a model that overfits?.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Hello, thanks for your answer. 'When cross validating, the model is not trained and evaluated on the same data.' I was referring to return_train_score in scikit-learn cross_validate. I'm using 3-split cv and getting 1.0 roc-auc on each train fold while test folds differ from 0.6 to 0.8. So mean cv_test-score is ~0.75 while mean cv_train score is 1.0 $\endgroup$
    – ike
    Commented Jun 26 at 20:37
  • $\begingroup$ Ok, that clears that part up then. I don't think the other results are very strange. Yes, your overfit, and you might get some benefit from using a less flexible model. $\endgroup$
    – Gijs
    Commented Jun 27 at 10:36
  • $\begingroup$ One more question. I'm using gradient boosting on decision tree and getting those metrics. Would you rather try to make it less complex by adjusting regularization and hyperparamteres or use more simple model but with more complex feature generation/engineering? For example, with logistic regression I'm getting ~0.6 on train and test without any complex feature engineering $\endgroup$
    – ike
    Commented Jun 27 at 13:27
  • $\begingroup$ If your goal is predictive perfomance then XGBoost will do well. Adjust tree depth and number of estimators. Feature engineering can always help. $\endgroup$
    – Gijs
    Commented Jun 28 at 7:13
1
$\begingroup$

But when I do model(best_parameters).fit(X_train, y_train) and then .predict_proba(X_train), I get ROC-AUC = 1.0.

Yes, this would mean that the model is fitting your training data perfectly. That is overfitting. The model can fit the training data perfectly, and the predictions are exactly the input of the data.

Also, during cross-validation, the train-folds metric is 1.0.

This is strange. When cross validating, the model is not trained and evaluated on the same data. So while the results of cross-validation may be a little bit higher than your final testing results, the difference is too high. But are you sure this is what you mean, I understand this to be contradicting

CV metric: ~0.75 ROC-AUC

Test: ~0.74 ROC-AUC

0.75 would make sense for the cross validation. That means that the model is very flexible, so it is can overfit on the training data. So conventional wisdom is to try to reduce overfitting by making it less flexible. This should bring the training error closer to the testing error.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.