1
$\begingroup$

I am working on a binary classification problem using balanced bagging random forest, neural networks and boosting techniques. my dataset size is 977 and class proportion is 77:23.

I had 61 features in my dataset. However, after lot of feature selection activities, I arrived at 5 features. But yes, these 5 features were identified using random forest estimator in RFECV, Borutapy etc. So, with 5 features, I thought that my Xgboost model will not overfit and provide me better performance in test set but still the Xgboost model overfits and produces poor results on test set. However, Random forest does similar performance on both train and test. Can help me understand why does this happen?

performance shown below for train and test

Random Forest - train data

enter image description here

Random Forest - test data

enter image description here

roc_auc for random forest - 81

Xgboost - train data

enter image description here

Xgboost - test data

enter image description here

roc_auc for xgboost - 0.81

Improve this question
$\endgroup$
7
  • $\begingroup$ Are you sure this is overfit or just "worse" results on the test set? But worse might be a stretch. I see the auroc is the same between gbm and rf. The confusion matrix is different. is the cutoff optimized for each model (RF and GBM)? Taking the default cutoff might not be optimal to the problem you are solving nor comparable across algorithms? $\endgroup$
    – Craig
    Commented Mar 4, 2022 at 11:13
  • $\begingroup$ Why is neural network tagged? The question says random forrests and xgboost. Amy I missing a step from the question? $\endgroup$
    – Craig
    Commented Mar 4, 2022 at 11:14
  • $\begingroup$ @craig - I used neural network to classifivstion as well but the results aren't shown here. Yes. How do you differenentiate overfittin from a worse test set performance? Yes, am using default threshold. While understand the use of custom thresholds, may I know why do you think that threshold has to be different from diff algo? $\endgroup$
    – The Great
    Commented Mar 4, 2022 at 22:12
  • $\begingroup$ If performance in test drops heavily when compared to train, I thought they are called ovsrfitting. How else should we understand this? $\endgroup$
    – The Great
    Commented Mar 4, 2022 at 22:13
  • $\begingroup$ There is no perfect definition of overfitting from my experiences. You mentioned "drops heavily". How much is heavily? Sometimes validation curves can show something but other times not. The threshold may need to be different on different algos since each algorithm is different. They get different observations "right" and "wrong". They have different space between the scores of the observations. If all algos gave the same answer, there would be no reason for multiple algos. Using the threshold specific to the cost/benefit you are solving for may show a different result. $\endgroup$
    – Craig
    Commented Mar 7, 2022 at 11:02

1 Answer 1

Reset to default
1
$\begingroup$

Overfitting not only depends on number of features but other hyperparametrs also. I will suggest you to do a extensive Hyperparameter tuning and use early stopping while training the final model. This may lead to better results.

Otherwise, without doing experimentation its very difficult to gauge the reason for overfitting

Improve this answer
$\endgroup$
2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.