L1 & L2 Regularization in Light GBM

Question

This question pertains to L1 & L2 regularization parameters in Light GBM. As per official documentation:

reg_alpha (float, optional (default=0.)) – L1 regularization term on weights.

reg_lambda (float, optional (default=0.)) – L2 regularization term on weights

I have seen data scientists using both of these parameters at the same time, ideally either you use L1 or L2 not both together.

While reading about tuning LGBM parameters I cam across one such case: Kaggle official GBDT Specification and Optimization Workshop in Paris where Instructors are ML experts. And these experts have used positive values of both L1 & L2 params in LGBM model. Link below (Ctrl+F 'search_spaces' to directly reach parameter grid in this long kernel)

http://www.kaggle.com/lucamassaron/kaggle-days-paris-gbdt-workshop

I have seen same in XGBoost implementations.

My question is why use both at the same time in LGBM/XGBoost.

Thanks.

Answer 1

First, note that in logistic regression, using both an L1 and an L2 penalty is common enough to have its own name: ElasticNet. (Perhaps see https://stats.stackexchange.com/q/184029/232706 .) So using both isn't unprecedented.

Second, XGBoost and LightGBM have quite a number of hyperparameters that overlap in their purpose. Tree complexity can be controlled by maximum depth, or maximum number of leaves, or minimum sample (count or weight) per leaf, or minimum criterion gain. Any combination of these might be optimal for some problem. Overfitting can also be combatted with the learning rate vs. number of trees (and early stopping), subsampling rates, and either of the regularization penalties.

Finally, since L1 regularization in GBDTs is applied to leaf scores rather than directly to features as in logistic regression, it actually serves to reduce the depth of trees. This in turn will tend to reduce the impact of less-predictive features, but it isn't so dramatic as essentially removing the feature, as happens in logistic regression. You might think of L1 regularization as more aggressive against less-predictive features than L2 regularization. But then it might make sense to use both: some L1 to punish the less-predictive features, but then also some L2 to further punish large leaf scores without being so harsh on the less-predictive features.

Toy example: https://github.com/bmreiniger/datascience.stackexchange/blob/master/trees_L1_reg.ipynb

Possibly useful:
https://github.com/dmlc/xgboost/blob/release_0.90/src/tree/split_evaluator.cc#L118
https://xgboost.readthedocs.io/en/latest/tutorials/model.html#model-complexity

Answer 2

In this medium post, you can find a concise and very clear explanation regarding these parameters https://medium.com/@gabrieltseng/gradient-boosting-and-xgboost-c306c1bcfaf5

Gabriel Tseng, Author of the blogpost: "These two regularization terms have different effects on the weights; L2 regularization (controlled by the lambda term) encourages the weights to be small, whereas L1 regularization (controlled by the alpha term) encourages sparsity — so it encourages weights to go to 0. This is helpful in models such as logistic regression, where you want some feature selection, but in decision trees we’ve already selected our features, so zeroing their weights isn’t super helpful. For this reason, I found setting a high lambda value and a low (or 0) alpha value to be the most effective when regularizing."

Answer 3

I have seen data scientists using both of these parameters at the same time, ideally either you use L1 or L2 not both together.

Shortly - use both to penalize too small weights by L1 and high weights (== outliers) by L2 to prevent overfitting.