# Understanding regularization

I'm currently trying to understand regularization for logistic regression.

So far, I'm not quite sure whether I really got it. Basically, the problem is that when we add an additional features to a model we might overfit the training set. This leads to an algorithm which perfectly matches the training set, but fails for future values (as it too exactly fits what has been there in the past).

So, this is why we introduce regularization. With this we decrease the impact of the features, so the model may produce an algorithm which does a (seemingly) worse job on the training set, but overall works better.

Is this correct?

For example, think of a polynomial $a_n x^n + a_{n-1} x^{n-1} + \dots + a_2 x^2 + a_1 x^1 + a_0 x^0$ which should fit 100 data points $(x_, y_i)$ where all $y_i$ were generated by one polynomial with some noise. Of course, you could always perfectly fit the model to the data making the MSE error
$$MSE = \sum_{(x_i, y_i)} (y_i - \text{predict}_{\text{Model}}(x_i))^2$$
be 0. But considering that there is noise, you might actually prefer a "simpler" model. One way to think about simplicity is having weights $a_i$ which are smaller. This is done by adding a regularization term to the error. One common regularization is $L_1$ regularization ($+ \sum_{1}^{100} | a_i |$), another one is $L_2$ regulariazion ($+ \sum_{1}^{100} a_i^2$).