In practice, which parameter do you typically tune first? Do you tune the learning rate (or step size) first? and then tune the total number of iterations? And how do you go about tuning these parameters?


1 Answer 1


Decision Tree-based algorithms can be complex and prone to overfitting. Depending on the data availability and various statistical metrics that give us an overall understanding of how the data is, it becomes important for us to do the right set of hyperparameter adjustments.

Since hyperparameter tuning/ optimization is a broad topic on its own, in the following subsections we are going to aim to get an overall understanding of some of the hyperparameter tuning and some important hyperparameters for both XGBoost and LightGBM.

Parameters in ML models

  • The objective of a typical learning algorithm is to find a function f that minimizes a certain loss over a dataset.
  • The learning algorithm produces f through the optimization of a training criteron with respect to a set of parameters.

Hyperparameters in ML models

  • Hyperparameters are parameters that are not directly learnt by the learning algorithm.
  • Hyperparameters are specified outside of the training procedure.
  • Hyperparameters control the capacity of the model, i.e., how flexible the model is to fit the data.
  • Prevent over-fitting.
  • Hyperparameters could have a big impact on the performance of the learning algorithm.
  • Optimal hyperparameter settings often differ for different datasets. Therefore they should be optimized for each dataset.

Hyperparameter Nature

  • Some hyperparameters are discrete: Number of estimators in ensemble models.
  • Some hyperparameters are continuous: Penalization coefficient, Number of samples per split.
  • Some hyperparameters are categorical: Loss (deviance, exponential), Regularization (Lasso, Ridge)


Different hyperparameter optimization strategies:

  • Manual Search
  • Grid Search
  • Random Search
  • Bayesian Optimization

XGBoost parameters

Here are the most important XGBoost parameters:

  1. n_estimators [default 100] – Number of trees in the ensemble. A higher value means more weak learners contribute towards the final output but increasing it significantly slows down the training time.
  2. max_depth [default 3] – This parameter decides the complexity of the algorithm. The lesser the value assigned, the lower is the ability for the algorithm to pick up most patterns (underfitting). A large value can make the model too complex and pick patterns that do not generalize well (overfitting).
  3. min_child_weight [default 1] – We know that an extremely deep tree can deliver poor performance due to overfitting. The min_child_weight parameter aims to regularise by limiting the depth of a tree. So, the higher the value of this parameter, the lower are the chances of the model overfitting on the training data.
  4. learning_rate/ eta [default 0.3] – The rate of learning of the model is inversely proportional to the accuracy of the model. Lowering the learning rate, although slower to train, improves the ability of the model to look for patterns and learn them. If the value is too low then it raises difficulty in the model to converge.
  5. gamma/ min_split_loss [default 0] – This is a regularization parameter that can range from 0 to infinity. Higher the value, higher is the strength of regularization, lower are the chances of overfitting (but can underfit if it’s too large). Hence, this parameter varies across all types of datasets.
  6. colsample_bytree [default 1.0] – This parameter instructs the algorithm on the fraction of the total number of features/ predictors to be used for a tree during training. This means that every tree might use a different set of features for prediction and hence reducing the chances of overfitting and also improving the speed of training as not all the features are being used in every tree. The value ranges from 0 to 1.
  7. subsample [default 1.0] – Similar to colsample_bytree, the subsample parameter instructs the algorithm on the fraction of the total number of instances to be used for a tree during training. This also reduces the chances of overfitting and improves training time.

Find more parameters here.

LightGBM parameters

Here are the most important LightGBM parameters:

  1. max_depth – Similar to XGBoost, this parameter instructs the trees to not grow beyond the specified depth. A higher value increases the chances for the model to overfit.
  2. num_leaves – This parameter is very important in terms of controlling the complexity of the tree. The value should be less than 2^(max_depth) as a leaf-wise tree is much deeper than a depth-wise tree for a set number of leaves. Hence, a higher value can induce overfitting.
  3. min_data_in_leaf – The parameter is used for controlling overfitting. A higher value can stop the tree from growing too deep but can also lead the algorithm to learn less (underfitting). According to LightGBM’s official documentation, as a best practice, it should be set to the order of hundreds or thousands.
  4. feature_fraction – Similar to colsample_bytree in XGBoost
  5. bagging_fraction – Similar to subsample in XGBoost

Find more parameters here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.