I am doing a regression and I want to use the regularizer that will be the most useful to get a sparse set of parameters.
Which regularizer should I use ? Cardinality? maximum value ? Sum of absolute values ? Euclidean norm ?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
The most common sparse regularizer is sum of absolute values (so-called Lasso regression). With carefully chosen penalty coefficient, it makes some of less useful parameters exactly zero.
Cardinality penalty exactly imposes sparsity, but it cannot be combined with gradient descent, and usually requires combinatorial optimization. Simply put, it is slow to apply.
Maximum value and Euclidean norm do not affect sparsity at all.
You can find more details in the wikipeda article.