Why is the L2 penalty squared but the L1 penalty isn't in elastic-net regression?

Question

There was some data set I worked with which I wanted to solve non negative least squares (NNLS) on and I wanted a sparse model. After a bit of experiementing I found that what worked the best for me was using the following loss function:

$$\min_{x \geq 0} ||Ax-b|| + \lambda_1||x||_2^2+\lambda_2||x||_1^2$$

Where the L2 squared penalty was implemented by adding white noise with a standard deveation of $\sqrt{\lambda_1}$ to $A$ (which can be showed to be equivelent to ridge regression in the expectation) and I implemented the L1 squared penalty by adding a row to $A$ with a constant value of $\sqrt{\lambda_2}$ and added a value of 0 to the end of $b$, which in the case of NNLS can be shown to be equivelent to L1 squared penalty.

That worked good for my purposes, but I know that usually in sparse regression models (for example elastic net or lasso regression) the L1 penalty is not squared, so it made me wonder if there could be a problem I'm missing with using squared L1 penalty? Is there any specific reason the L1 penalty is not squared but the L2 penalty is squared in elastic net regression?

Answer 1

Thinking about it more made me realise there is a big downside to L1 squared penalty that doesn't happen with just L1 or L2 squared.

The downside is that each variable even if it's completely orthogonal to all the other variariables (i.e., uncorrelated) gets influanced by the other variables in the L1 squared penalty because the penalty is no longer a sum of penalties for each variable (like in the l1 or l2 squared), that means that uncorrelated variables gets correlated which is a problem, in general we almost never want uncorrelated variables to influence each other.

A simple case to show why this is a problem is the following:

Let's say we have a a matrix $A_1$ and a desired vector $b_1$ and then we have another matrix $A_2$ and a desired vector $b_2$. If we solved lasso/ridge on each of them separately we'll get the same answer as if we solved the problem on the matrix:

$$A_1 \ \ 0\\ 0 \ \ A_2$$

And the desired vector $$b_1\\ b_2 $$

But in the case of squared L1 penalty this will not be the case, because even though the first half of the colomns are uncorrelated to the second half they will get correlated by the penalty.

Sorry for bad English.