I am trying to understand regularisation for logistic regression currently, but I am not sure I get it. I understand the issue of overfitting when there are relatively too many features, and I get that you would like to limit the impact of some of these superfluous features, but you are doing in regularisation is impacting all features.
And this brings me to my next point, which is that by putting a limit on weights, you can still construct the same hyperplane. As long as you have the same ratio between the weights $w_i$ for $i \in \{1,2,..,n\}$, the "angle" of the hyperplane would still be the same, and then you can simply "adjust" it's position by the intercept $w_0$.
Many lecturers and authors use the complex polynomial example with additional terms $f(x_i)$ that make the decision boundary non-linear to illustrate that using regularisation will make this boundary more linear, and thus less prone to overfitting, but then again, you could plot $f(x_i)$ on a separate axis, making decision boundary a hyperplane, and then you could fit the exact same decision boundary with smaller weights, as long as the ratios are conserved.
Thus, if you can make the exact same hypothesis with smaller sum of squares of weights $\sum w_i^2$ (that is with smaller regularisation term), what is the point of regularisation? Or in other words, as there obviously exists empirical evidence showing that regularisation works, how does it work? Is there a good proof for it, or some good (and less vague) intuition?