Good explanation for why regularisation works

Question

I am trying to understand regularisation for logistic regression currently, but I am not sure I get it. I understand the issue of overfitting when there are relatively too many features, and I get that you would like to limit the impact of some of these superfluous features, but you are doing in regularisation is impacting all features.

And this brings me to my next point, which is that by putting a limit on weights, you can still construct the same hyperplane. As long as you have the same ratio between the weights $w_i$ for $i \in \{1,2,..,n\}$, the "angle" of the hyperplane would still be the same, and then you can simply "adjust" it's position by the intercept $w_0$.

Many lecturers and authors use the complex polynomial example with additional terms $f(x_i)$ that make the decision boundary non-linear to illustrate that using regularisation will make this boundary more linear, and thus less prone to overfitting, but then again, you could plot $f(x_i)$ on a separate axis, making decision boundary a hyperplane, and then you could fit the exact same decision boundary with smaller weights, as long as the ratios are conserved.

Thus, if you can make the exact same hypothesis with smaller sum of squares of weights $\sum w_i^2$ (that is with smaller regularisation term), what is the point of regularisation? Or in other words, as there obviously exists empirical evidence showing that regularisation works, how does it work? Is there a good proof for it, or some good (and less vague) intuition?

Answer 1

The "same angle hyperplane" does not have the same cost. It is the same decision boundary as you describe it, but perpendicular distances to it are larger wrt the norm of the weights. In effect with higher weights in the same ratio (i.e. without any regularisation effect), the classifier will be more confident in all of its decisions. That means the classifier will be more sensitive to getting as many observations as possible in the training set on the "right" side of the boundary. In turn this makes it sensitive to noise in the observations.

Your estimated probability for being in the positive class is:

$$p(y=1|X) = \frac{1}{1+e^{-W^TX}}$$

This includes $w_0$ and fixed value 1 for $x_0$. If you take the midpoint, decision line where $W^TX$ is zero (and the output is at the threshold value 0.5), that defines your decision hyperplane in $X$ space.

When $W$ has the same factors but a larger norm with $w_0$ compensating to make the same hyperplane, then $X$ values that were on the decision hyperplane still give threshold value of 0.5. However, $X$ values away from the hyperplane will deviate more strongly. If instead of 0 you had $W^TX=1.0$ and doubled the weights keeping the same hyperplane, you would get $W^TX=2.0$ for that example. Which changes your confidence from 0.73 to 0.88.

The usual cost function without regularisation for logistic regression with example vectors $X_j$ and targets $y_j$ is:

$$J = - \sum_{\forall j} y_jlog(\frac{1}{1+e^{-W^TX_j}}) + (1 -y_j)(1 - log(\frac{1}{1+e^{-W^TX_j}}))$$

The cost is more sensitive to distances from the hyperplane for larger weight values. Looking your example for the imaginary item (with 0.73 or 0.88 confidence), when the categorisation is correct (i.e. y=1) the score would improve by 0.19 for that example if the weights doubled. When the categorisation is wrong (y=0) then the score would worsen by 0.81. In other words for higher weights, with same weight ratio, the same miscategorisations are punished more than correct categorisations are rewarded.

When training, weights will converge to specific balanced weight vector for the minimum cost, not to a specific ratio that forms a "best decision hyperplane". That's because the hyperplane does not correspond to a single value of the cost function.

You can demonstrate this effect. Train a logistic regression classifier - without any regularisation to show it has nothing to do with that. Take the weight vector and multiply by some factor e.g. 0.5.Then re-train starting with those weights. You will end up with the same weights as before. The cost function minimum clearly defines specific weight values, not a ratio.

When you add regularisation, that changes the cost and how the weights will converge. Higher regularisation in effect makes the classifier prefer a boundary with lower confidence in all its predictions, it penalises "near misses" less badly because the weights are forced down where possible. When viewed as a hyperplane, the boundary will likely be different.