Why does feature selection matter if your model has L1 regularization?

Question

I've been tinkering around with boosted trees, and I saw that for common libraries there is a parameter you can set to determine L1 regularization. I doubled my original feature set to around 130 features so that it contained some noise, some new useful features, and a high degree of multicollinearity and the model performed noticeable worse across several experiments.

I was wondering why this would be the case if L1 shrinks induces sparsity in the feature set. I would expect the model to discard the noise, and keep the new useful features, resulting in better performance but this isn't the case. Why is that?

Answer 1

This is because L1 regularization does not do the exact same thing in a Linear Regression as it does in a Tree Model.

L1 is aiming at doing the same thing in both models, to improve performance. However, they do not accumulate in the same way.

In the linear regression the coefficients are penalized toward zero, so you can effectively remove a variable from the set altogether.

In a boosted tree algorithm, the leaves(nodes) residual values are evaluated and an adjustment (penalty) is passed forward to the NEXT TREE. This proceeds iteratively until the final outcome. But what is passed forward is based on splits and there are a lot of them to choose from in wide datasets.

It is likely it does work toward zero in your boosted tree, but with many, many variables and lots of noise, it is possible that there are a few big splits which resolve a lot of entropy early on that are just happenstansical noise and they look like information. And also depending on the size of the forest and the depth of the model, you may not have reached the optimal model.

You asked about the feature set, in particular the linear regression can effectively remove the whole variable and the entire variable is evaluated in the model. In the tree, specific splits at specific points in a variable are evaluated and penalized forward at each split. The penalty therefore does not accrue in the same way.