I have heard regularization helps to get rid of outliers, how so? 'My intuition is, regularization shrinks parameter or even make it zero, and hence large value will have less effect on overall result'. Could you shed some more light on it?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
You don't get rid of the actual outliers (no data reduction). But statistical methods can be robust against outliers.
Robust statistics are statistics with good performance for data drawn from a wide range of probability distributions, especially for distributions that are not normal. Robust statistical methods have been developed for many common problems, such as estimating location, scale, and regression parameters. One motivation is to produce statistical methods that are not unduly affected by outliers.
Source: wikipedia.
So, L-1 regularization is robust against outliers as it uses the absolute value between the estimated outlier and the penalization term.
Whereas, L2-regularization is not robust against outliers as the squared terms blow up the differences between estimation and penalization.
-
$\begingroup$ I think I would argue that $L_2$ regularization is more robust to outliers than $L_1$ regularization, not less? Or, more precisely, that including $L_2$ regularization to your loss function ultimately makes your estimated model more robust to presence of extreme outliers in the data, than $L_1$ regularization would. The penalty used by $L_1$ regularization is less sensitive to outliers, but that means it punishes the effects of those outliers on the training less severely, so ultimately it helps less to make your model robust. $\endgroup$ Oct 5 at 7:45