Understanding XG Boost Training (Multi class classification)

Question

I have been working with XG boost for classification (multi class classification : 6 classes) I use 5 fold CV to train and validate my model.

Please refer to the paramters, which i had used in my model.

 params    = {"objective": 'multi:softprob', "eta": 0.1, "max_depth": 7,
               "min_child_weight": 4,"silent": 1, "subsample": 0.8,
               "colsample_bytree": 0.8, "num_class" : 6, "gamma" : 0, 
               "eval_metric" : 'merror', "seed": 0}

I plotted the training and testing error for each fold in a 5 fold CV.

Questions:

1. What can I understand/interpret from the training & test loss graph ?
2. Training error reduces to zero, but the testing error reduces over a period & become idle.
   • I am not sure whether the model is overfitting ?
   • How can I reduces the error b/w training & testing, only through hyperparam tunning or this is how the XG Boost model works ?

Answer 1

What can I understand/interpret from the training & test loss graph?

This checking out the quality of the model. If the train and test set loss decreasing according to the number of epoch in the same way(i.e plots should overlap each other), that means that model is good. Otherwise, we have a problem. In your graph plots fastly separates(from x-value moreover 10 or 20). In this case we could say that's overfitting.

I am not sure whether the model is overfitting ?

It's typical overfitting.

How can I reduces the error b/w training & testing, only through hyperparam tunning or this is how the XG Boost model works ?

• More data.
• Feature selection.
• Cross-validation.
• Wide range hyperparameters to explore.
• Random search.

Answer 2

Assuming your x axis is nrounds(Or ntrees) parameter,

1. XGB is an ensemble of many many trees built on top of one another. Your XAxis indicates how many trees have been used. Consider 2 points at x = 100 and x= 200,

When you had 100 trees the train and test loss were close to .15 and 0.26, but on building 100 more trees on top of this train loss reduced to 0.08 and test dropped to 0.24.

2. It is definitely overfitting, infact it looks like growing any tree beyond 200 is total waste. But you will have to try tuning the following parameters to add regularization, to reduce gap b/w train-test loss further

   • num_round - This is number of tree parameter, only grow more trees if you are seeing improvement in validation loss. And if that improvement in validation loss is worth having many more trees. I would typically first keep this large just to look at how loss is dropping and pick that num_round above which it takes too many more trees for validation loss to drop significantly

   • max_depth - Reduce this, remember XGB is an ensemble of weak trees. You individual trees must be very shallow, So I typically tweak this param b/w 3-6. This is maximum length of your tree. This is the first param I would tweak.

   • lambda and alpha - Increase these params. These add L1 and L2 regularization. It can take any positive value and more you increase this, lesser is the chance of over-fitting.

   • gamma and min_child_weight - https://medium.com/data-design/xgboost-hi-im-gamma-what-can-i-do-for-you-and-the-tuning-of-regularization-a42ea17e6ab6 Explained well here.

   • eta Not to forget this. You can use this in conjunction with num_rounds. Reducing will result in your model learning slowly w.r.t num_rounds. Typically this is tuned using gridsearch/bayesian optimization to get best results.

All parameters: https://xgboost.readthedocs.io/en/latest/parameter.html#