# Overfitting XGBoost

I try to classify data from a dataset of 315 lines and 17 (real data) features (315x17). The target value is either "good" or "bad" (binary classification).

I used XGBoost to classify these data, but I get to much overfitting. I do cross validation using the logloss to evaluate the performance.

Green : logloss validation curve

Red : logloss training curve

X-axis : nrounds (max number of boosting iterations)

The first model (first picture was generated with :

bst.res <- xgb.cv(nfold = 4,
data = matrix_training_set,
label = training_set_label,
eta = 0.1,
# minimum sum of instance weight(hessian) needed in a child
min_child_weight = 4,
# maximum depth of a tree
max_depth = 14,
objective = "binary:logistic",
#L2 parameter
lambda = 1.1,
# L1 parameter
alpha = 0.5,
gamma = 0,
eval_metric = "logloss",
nrounds = 2000,
verbose = TRUE,
subsample = 0.7,
print_every_n = 10,
early_stop_round = 10)


Then, I tried (second picture) :

bst.res <- xgb.cv(nfold = 4,
data = matrix_training_set,
label = training_set_label,
eta = 0.01,
# minimum sum of instance weight(hessian) needed in a child
min_child_weight = 4,
# maximum depth of a tree
max_depth = 6,
objective = "binary:logistic",
#L2 parameter
lambda = 1.1,
# L1 parameter
alpha = 0.5,
gamma = 1,
eval_metric = "logloss",
nrounds = 2000,
verbose = TRUE,
subsample = 0.7,
print_every_n = 10,
early_stop_round = 10)


Basically, it is possible to reduce overfitting by changing max_depth, min_child_weight , gamma, subsample.

Below, again, I reduce the overfitting (3rd picture) :

bst.res <- xgb.cv(nfold = 4,
data = matrix_training_set,
label = training_set_label,
eta = 0.01,
# minimum sum of instance weight(hessian) needed in a child
min_child_weight = 7,
# maximum depth of a tree
max_depth = 6,
objective = "binary:logistic",
#L2 parameter
lambda = 1.1,
# L1 parameter
alpha = 0.5,
gamma = 5,
eval_metric = "logloss",
nrounds = 2000,
colsample_bytree = 0.7,
verbose = TRUE,
subsample = 0.3,
print_every_n = 10,
early_stop_round = 10)


Now it seems that the model is biased (red and green are indeed really close, take care of the scale).

I also tried grid searching, but the validation curve always stand really high (over 0.5 for logloss and less than 0.62 for AUC).

Now I'm wondering if I should create a model more biased (make the trees simplier), and then add more data to reduce this bias.

Any ideas to make this green curve lower ? Is that possible that there is no possible correlation between my target ("good" or "bad") and my features, which means that it becomes impossible to create a classifier from these data ?

• Did you check the variable importance of models after ~50 rounds? If it peaks for few features, then check your features. If it is almost equally distributed, there is a chance that your labels have no correlation with data. Maybe some bugs in selecting and merging labels?! Oct 12, 2017 at 14:43
• Have you tried comparing the results you posted here to baseline results produced from using a simpler model built using default parameter values? You could even build a model without CV to use for comparison. Oct 17, 2017 at 1:41
• You can try grid search to see the model that does better on the validation set, right? Apr 20, 2018 at 7:42

What I see is that there is a big variance in your data. This means that without a sufficient amount of samples (bigger than 315) it will be simply impossible to describe the dataset for any model you can think.

But, if the variance is not so big:

1. You can try a data augmentation technique.

2. Have you shuffled the data before separating it in training/validation set? Maybe is a banal operation, but if in the validation set you have samples which are too much different from the training set, they will be never correctly predicted.

3. Are the classes quite unbalanced? If so, maybe you have too few samples of one of them, and in that case you can try some upsamplig technique.

Try a random forest, or a regularized model like lasso instead. They're less susceptible to overwriting. RF has the added benefit that you can use the OOB estimate to evaluate performance, so you don't have to slice up your tiny dataset into train and test.

Boosting generally isn't a good choice for small data and tends to overfit as you've observed. If you insist on boosting, robust boosting is a modified variant that is specifically designed to work better on small datasets.