Understanding regularization

Question

I'm currently trying to understand regularization for logistic regression.

So far, I'm not quite sure whether I really got it. Basically, the problem is that when we add an additional features to a model we might overfit the training set. This leads to an algorithm which perfectly matches the training set, but fails for future values (as it too exactly fits what has been there in the past).

So, this is why we introduce regularization. With this we decrease the impact of the features, so the model may produce an algorithm which does a (seemingly) worse job on the training set, but overall works better.

Is this correct?

Answer 1

That's correct. Without regularization your model would fit to an irrelevant noise present in your dataset. It means that training set will fit better but the overall predictive power will decrease. This is a good article about overfitting and regularization.

Also you can get some intuition by watching these images: Without regularization With regularization You can clearly see that the shape at the 1st image has problems with prediction at the upper left corner but it perfectly fits to the training points.

Answer 2

Yes, that is correct.

For example, think of a polynomial $a_n x^n + a_{n-1} x^{n-1} + \dots + a_2 x^2 + a_1 x^1 + a_0 x^0$ which should fit 100 data points $(x_, y_i)$ where all $y_i$ were generated by one polynomial with some noise. Of course, you could always perfectly fit the model to the data making the MSE error

$$MSE = \sum_{(x_i, y_i)} (y_i - \text{predict}_{\text{Model}}(x_i))^2$$

be 0. But considering that there is noise, you might actually prefer a "simpler" model. One way to think about simplicity is having weights $a_i$ which are smaller. This is done by adding a regularization term to the error. One common regularization is $L_1$ regularization ($+ \sum_{1}^{100} | a_i |$), another one is $L_2$ regulariazion ($+ \sum_{1}^{100} a_i^2$).