Interpretation of tuning parameters (shrinkage and nrounds) in XGBoost

Question

I'm using XGBoost for a multiclass classification problem in R. I'm trying different combinations of the shrinkage parameter and number of iterations to try and settle on optimal values for these parameters (using cross-validation). However, I'm having trouble with exactly interpreting the significance of the parameters. The XGBoost algorithm isn't completely a black box for me - I know how boosting works at a high-level.

The questions I have though are as follows:

1. Does a low shrinkage (e.g. 0.01) and high nrounds (e.g. 500) yield the same results as a high shrinkage (e.g. 0.3) and low nrounds (e.g. 50)?

2. Does nrounds correspond to the number of weak classifiers being fit?

3. The xgb.cv function in R returns test mean errors and standard deviations for each of the iterations. For the nth iteration, is this the mean over all iterations less than equal to n over all the folds? Apologies if I'm interpreting this incorrectly. Also, what is the significance of the standard deviation here?

4. Is there some blog post/link I can refer to for a comprehensive guide on tuning XGBoost parameters?

Answer 1

From my understanding (and I am no expert!)

Point 1

Let $ \hat f_k $ be a weak learner or one tree in the model, then we would make a prediction as the $$\hat y_i = \sum_{i=0}^k \hat f_k(x) $$ with $ (x) $ being a particular feature The algorithm learns in an additive way so $$f_0(x) + f_1(x) + f_2(x) .... f_k(x) = \hat y_i $$ So adding a shinkage parameter $ \delta $ will penalise the model for each additional tree (or weak learner) you add to it, which looks like $ \delta f_k $. So having a low shinkage $ \delta $ would penalise the overall prediction score quite a lot.

Having a shrinkage score of 0.3 vs 0.01 with some $ (x) $ variable scores from each tree $f_k$;

Model 1: $ 0.3(2.1) + 0.3(2.9) + 0.3(3.4) + 0.3(3.5) = 3.57$

Model 2: $ 0.01(2.1) + 0.01(2.9) + 0.01(3.4) + 0.01(3.5) = 0.119 $

After four trees, so according to my understanding Model 2 is going to need many more trees to get to the same result as Model 1. However having a low number of rounds will likely miss important structural information in the model. I understand it this way.

Shrinkage extremely high and low, 1 and 0.01

Model 1x: $ 1(2.1) + 1(2.9) + 1(3.4) + 1(3.5) = 11.9 $

Model 2x: $ 0.01(2.1) + 0.01(2.9) + 0.01(3.4) + 0.01(3.5) = 0.119 $

The difference here being that Model 2x can capture 3 decimal places in its increments, whereas Model 1x goes up in increments of 0.1. So if we choose Model 1x then its possible to overshoot the optimal number of trees. If we were to increase the number of trees for Model 2x to 500 or 1000 then its possible that we would overfit the data, alternatively if we were to reduce the number of trees for Model 1x we would underfit the data. So this is where the cross validation and search for optimal parameters comes in.

So to answer point 1. Theoretically I understand yes, the numbers will converge, however this would lead to potential overfitting issues and a balance is necessary.

Point 2) nrounds to me, means the number of additional trees or weak learners added to the model.

At $ nround = 0 $ we have;

$$ \hat y_i^0 = 0$$

at $nround = 1$ we have;

$$ \hat y_i^1 = \hat y_t^0 + \delta f_1(x)$$

so $ \hat y_i $ adds the previous weak learners to its new prediction

then $ nround = 2 $ we have

$$ \hat y_i^2 = \hat y_i^1 + \delta f_2(x) $$

So at each $nround$ a new weak learner is added to the prediction.

Point 3) Here for me the mean evaluation metric is the mean score over k-fold cross validation. If you set k-folds to 10 then it would run the model 10 times and produce just the mean of these scores, the same with standard deviation scores.

Point 4) Theres many places to read about xgboost tuning, I have visited many of these websites countless times here

One really cool piece of code I am using from here. Although my code now has expanded this for most of the parameters of XGBoost and for an AUC evaluation metric not RMSE. I can post it if you are using AUC for evaluation.

I am still no expert but I am trying to answer some of the questions that you have myself, so I post my understanding of what I know, sure somebody will correct me :)

searchGridSubCol <- expand.grid(subsample = c(0.5, 0.75, 1), 
                                colsample_bytree = c(0.6, 0.8, 1))
ntrees <- 100

#Build a xgb.DMatrix object
DMMatrixTrain <- xgb.DMatrix(data = yourMatrix, label = yourTarget)

rmseErrorsHyperparameters <- apply(searchGridSubCol, 1, function(parameterList){

    #Extract Parameters to test
    currentSubsampleRate <- parameterList[["subsample"]]
    currentColsampleRate <- parameterList[["colsample_bytree"]]

    xgboostModelCV <- xgb.cv(data =  DMMatrixTrain, nrounds = ntrees, nfold = 5, showsd = TRUE, 
                           metrics = "rmse", verbose = TRUE, "eval_metric" = "rmse",
                           "objective" = "reg:linear", "max.depth" = 15, "eta" = 2/ntrees,                               
                           "subsample" = currentSubsampleRate, "colsample_bytree" = currentColsampleRate)

    xvalidationScores <- as.data.frame(xgboostModelCV)
    #Save rmse of the last iteration
    rmse <- tail(xvalidationScores$test.rmse.mean, 1)

    return(c(rmse, currentSubsampleRate, currentColsampleRate))

})