4
$\begingroup$

This question pertains to L1 & L2 regularization parameters in Light GBM. As per official documentation:

reg_alpha (float, optional (default=0.)) – L1 regularization term on weights.

reg_lambda (float, optional (default=0.)) – L2 regularization term on weights

I have seen data scientists using both of these parameters at the same time, ideally either you use L1 or L2 not both together.

While reading about tuning LGBM parameters I cam across one such case: Kaggle official GBDT Specification and Optimization Workshop in Paris where Instructors are ML experts. And these experts have used positive values of both L1 & L2 params in LGBM model. Link below (Ctrl+F 'search_spaces' to directly reach parameter grid in this long kernel)

http://www.kaggle.com/lucamassaron/kaggle-days-paris-gbdt-workshop

I have seen same in XGBoost implementations.

My question is why use both at the same time in LGBM/XGBoost.

Thanks.

$\endgroup$
2
$\begingroup$

First, note that in logistic regression, using both an L1 and an L2 penalty is common enough to have its own name: ElasticNet. (Perhaps see https://stats.stackexchange.com/q/184029/232706 .) So using both isn't unprecedented.

Second, XGBoost and LightGBM have quite a number of hyperparameters that overlap in their purpose. Tree complexity can be controlled by maximum depth, or maximum number of leaves, or minimum sample (count or weight) per leaf, or minimum criterion gain. Any combination of these might be optimal for some problem. Overfitting can also be combatted with the learning rate vs. number of trees (and early stopping), subsampling rates, and either of the regularization penalties.

Finally, since L1 regularization in GBDTs is applied to leaf scores rather than directly to features as in logistic regression, it actually serves to reduce the depth of trees. This in turn will tend to reduce the impact of less-predictive features, but it isn't so dramatic as essentially removing the feature, as happens in logistic regression. You might think of L1 regularization as more aggressive against less-predictive features than L2 regularization. But then it might make sense to use both: some L1 to punish the less-predictive features, but then also some L2 to further punish large leaf scores without being so harsh on the less-predictive features.

Toy example: https://github.com/bmreiniger/datascience.stackexchange/blob/master/trees_L1_reg.ipynb

Possibly useful:
https://github.com/dmlc/xgboost/blob/release_0.90/src/tree/split_evaluator.cc#L118
https://xgboost.readthedocs.io/en/latest/tutorials/model.html#model-complexity

$\endgroup$
  • $\begingroup$ Thanks Ben, this is super helpful. The last paragraph dispels my confusion: L1/L2 treats leaves not features and L1 is aggressive against insignificant features than L2 make my understanding better. $\endgroup$ – Vikrant Arora Aug 13 at 6:13
2
$\begingroup$

In this medium post, you can find a concise and very clear explanation regarding these parameters https://medium.com/@gabrieltseng/gradient-boosting-and-xgboost-c306c1bcfaf5

Gabriel Tseng, Author of the blogpost: "These two regularization terms have different effects on the weights; L2 regularization (controlled by the lambda term) encourages the weights to be small, whereas L1 regularization (controlled by the alpha term) encourages sparsity — so it encourages weights to go to 0. This is helpful in models such as logistic regression, where you want some feature selection, but in decision trees we’ve already selected our features, so zeroing their weights isn’t super helpful. For this reason, I found setting a high lambda value and a low (or 0) alpha value to be the most effective when regularizing."

$\endgroup$
  • 1
    $\begingroup$ That's helpful but question remains why positive values of both L1 & L2 together. $\endgroup$ – Vikrant Arora Aug 10 at 10:07
  • $\begingroup$ @VikrantArora, indeed, that author appears to mostly suggest not using alpha. $\endgroup$ – Ben Reiniger Aug 12 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.