Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Conducting a linear regression model using a loss function, why should I use $L_1$ instead of $L_2$ regularization?
Is it better at preventing overfitting? Is it deterministic (so always a unique solution)? Is it better at feature selection (because producing sparse models)? Does it disperse the weights among the features?
Basically, we add a regularization term in order to prevent the coefficients to fit so perfectly to overfit.
The difference between L1 and L2 is L1 is the sum of weights and L2 is just the sum of the square of weights.
L1 cannot be used in gradient-based approaches since it is not-differentiable unlike L2
L1 helps perform feature selection in sparse feature spaces.Feature selection is to know which features are helpful and which are redundant.
The difference between their properties can be summarized as :
L2 has one very important advantage to L1, and that is invariance to rotation and scale.
This is especially important in geographical / physical application.
Say your technician accidentally installed your sensor in 45 degree angel, L1 would be affected, while L2 (Euclidean distance) would remain the same.
