4
$\begingroup$

I'm trying to tune some parameters in XGBoost and read a lot about "...makes to model more conservative". Can somebody explain me what the word conservative means in this case?

I can imagine that it learns slower (more similar observations needed) and is less prone to overfitting? I'm not sure if my assumption is correct though.

Example from the docu:

eta [default=0.3, alias: learning_rate]
Step size shrinkage used in update to prevents overfitting. After each boosting step, we can directly get the weights of new features, and eta shrinks the feature weights to make the boosting process more conservative.

gamma [default=0, alias: min_split_loss]
Minimum loss reduction required to make a further partition on a leaf node of the tree. The larger gamma is, the more conservative the algorithm will be.

lambda [default=1, alias: reg_lambda]
L2 regularization term on weights. Increasing this value will make model more conservative.

and many more..

$\endgroup$

1 Answer 1

3
$\begingroup$

As you suggest, and although the term conservative might be confusing, it means having a less complex model by reducing a possible overfitting.

Parameters like lambda (L2 regularization) seem to make clear that, in this context, conservativeness means having a model that, although it could be more adjusted to your training set trying to capture all the possible information from your data and be more sensitive to the class of interest, it provides you a more "balanced" model so you have a lower variance, having more confidence when applying it to new data in the inference phase, trying to minimize the generalization error. A refresher on this concepts.

$\endgroup$
1
  • $\begingroup$ Thank you, that is what i expected. But it is good to have a confirmation! $\endgroup$
    – tturbo
    Nov 7, 2022 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.