I am using XGBoost regressor to train my model for 322 rows of data and the train and test split is as follows: ((257, 9), (257,), (65, 9), (65,))

I am using the following parameters for hyper-parameter tuning:

{'max_depth': 3,
 'min_child_weight': 6,
 'eta': 0.3,
 'subsample': 0.9,
 'colsample_bytree': 0.7,
 'objective': 'reg:linear',
 'eval_metric': 'rmse',
 'reg_lambda': 0,
 'reg_alpha': 0.5,
 'gamma': 0}

I am getting the following results:

Train results:

MAE =  43.95317769328908
RMSE =  69.32233101307436
R2 score =  0.7500463354991436


Test results : 

MAE =  51.21307032658503
RMSE =  79.65759750390318
R2 score =  0.6569142423871053

What are the drawbacks of training XGBoost model on such a small dataset? I know about overfitting, but I can control it to some extend with regularization.


The one other problem you have because of this is you are stuck with just this solution and this may not be the best one. Yes, you can use regularization but still, what if some other better solution exists than this. Since, in boosted trees, you aim is to reduce your errors, each iteration of boosting tries to overfit on the examples which the last iteration predicted wrong.

But there is a way to confirm if this is the best solution. Do cross-validation. Or try LightGBM and CatBoost also, since, they are also good libraries for the same purpose. And finally, you combine all three of them to come with an ensemble that will you the best of the three worlds.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.