Overfitting XGBoost

Question

I try to classify data from a dataset of 315 lines and 17 (real data) features (315x17). The target value is either "good" or "bad" (binary classification).

I used XGBoost to classify these data, but I get to much overfitting. I do cross validation using the logloss to evaluate the performance.

Green : logloss validation curve

Red : logloss training curve

X-axis : nrounds (max number of boosting iterations)

The first model (first picture was generated with :

bst.res <- xgb.cv(nfold = 4,
               data = matrix_training_set,
               label = training_set_label,
               eta = 0.1,
               # minimum sum of instance weight(hessian) needed in a child 
               min_child_weight = 4,
               # maximum depth of a tree 
               max_depth = 14,
               objective = "binary:logistic",
               #L2 parameter 
               lambda = 1.1,
               # L1 parameter 
               alpha = 0.5, 
               gamma = 0,
               eval_metric = "logloss",
               nrounds = 2000,
               verbose = TRUE,
               subsample = 0.7,
               print_every_n = 10,
               early_stop_round = 10)

Then, I tried (second picture) :

bst.res <- xgb.cv(nfold = 4,
               data = matrix_training_set,
               label = training_set_label,
               eta = 0.01,
               # minimum sum of instance weight(hessian) needed in a child 
               min_child_weight = 4,
               # maximum depth of a tree 
               max_depth = 6,
               objective = "binary:logistic",
               #L2 parameter 
               lambda = 1.1,
               # L1 parameter 
               alpha = 0.5, 
               gamma = 1,
               eval_metric = "logloss",
               nrounds = 2000,
               verbose = TRUE,
               subsample = 0.7,
               print_every_n = 10,
               early_stop_round = 10)

Basically, it is possible to reduce overfitting by changing max_depth, min_child_weight , gamma, subsample.

Below, again, I reduce the overfitting (3rd picture) :

bst.res <- xgb.cv(nfold = 4,
               data = matrix_training_set,
               label = training_set_label,
               eta = 0.01,
               # minimum sum of instance weight(hessian) needed in a child 
               min_child_weight = 7,
               # maximum depth of a tree 
               max_depth = 6,
               objective = "binary:logistic",
               #L2 parameter 
               lambda = 1.1,
               # L1 parameter 
               alpha = 0.5, 
               gamma = 5,
               eval_metric = "logloss",
               nrounds = 2000,
               colsample_bytree = 0.7,
               verbose = TRUE,
               subsample = 0.3,
               print_every_n = 10,
               early_stop_round = 10)

Now it seems that the model is biased (red and green are indeed really close, take care of the scale).

I also tried grid searching, but the validation curve always stand really high (over 0.5 for logloss and less than 0.62 for AUC).

Now I'm wondering if I should create a model more biased (make the trees simplier), and then add more data to reduce this bias.

Any ideas to make this green curve lower ? Is that possible that there is no possible correlation between my target ("good" or "bad") and my features, which means that it becomes impossible to create a classifier from these data ?

Answer 1

What I see is that there is a big variance in your data. This means that without a sufficient amount of samples (bigger than 315) it will be simply impossible to describe the dataset for any model you can think.

But, if the variance is not so big:

1. You can try a data augmentation technique.

2. Have you shuffled the data before separating it in training/validation set? Maybe is a banal operation, but if in the validation set you have samples which are too much different from the training set, they will be never correctly predicted.

3. Are the classes quite unbalanced? If so, maybe you have too few samples of one of them, and in that case you can try some upsamplig technique.

Answer 2

Try a random forest, or a regularized model like lasso instead. They're less susceptible to overwriting. RF has the added benefit that you can use the OOB estimate to evaluate performance, so you don't have to slice up your tiny dataset into train and test.

Boosting generally isn't a good choice for small data and tends to overfit as you've observed. If you insist on boosting, robust boosting is a modified variant that is specifically designed to work better on small datasets.