# why my boosting model overfits even with just 4 features out of 61?

I am working on a binary classification problem using balanced bagging random forest, neural networks and boosting techniques. my dataset size is 977 and class proportion is 77:23.

I had 61 features in my dataset. However, after lot of feature selection activities, I arrived at 5 features. But yes, these 5 features were identified using random forest estimator in RFECV, Borutapy etc. So, with 5 features, I thought that my Xgboost model will not overfit and provide me better performance in test set but still the Xgboost model overfits and produces poor results on test set. However, Random forest does similar performance on both train and test. Can help me understand why does this happen?

performance shown below for train and test

Random Forest - train data

Random Forest - test data

roc_auc for random forest - 81

Xgboost - train data

Xgboost - test data

roc_auc for xgboost - 0.81

• Are you sure this is overfit or just "worse" results on the test set? But worse might be a stretch. I see the auroc is the same between gbm and rf. The confusion matrix is different. is the cutoff optimized for each model (RF and GBM)? Taking the default cutoff might not be optimal to the problem you are solving nor comparable across algorithms? Mar 4 at 11:13
• Why is neural network tagged? The question says random forrests and xgboost. Amy I missing a step from the question? Mar 4 at 11:14
• @craig - I used neural network to classifivstion as well but the results aren't shown here. Yes. How do you differenentiate overfittin from a worse test set performance? Yes, am using default threshold. While understand the use of custom thresholds, may I know why do you think that threshold has to be different from diff algo? Mar 4 at 22:12
• If performance in test drops heavily when compared to train, I thought they are called ovsrfitting. How else should we understand this? Mar 4 at 22:13
• There is no perfect definition of overfitting from my experiences. You mentioned "drops heavily". How much is heavily? Sometimes validation curves can show something but other times not. The threshold may need to be different on different algos since each algorithm is different. They get different observations "right" and "wrong". They have different space between the scores of the observations. If all algos gave the same answer, there would be no reason for multiple algos. Using the threshold specific to the cost/benefit you are solving for may show a different result. Mar 7 at 11:02