I read a lot of articles online about how regularization works and most of them just show the equations with regularization terms but did not use example numbers to explain how the coefficient values change as lambda increase.

For example: L1 Regularization

theory states that When Lambda is large, coefficients tend to 0.

X =

enter image description here

How is the coefficients(beta i assume) value changing when lambda changes? Is it because X a fixed value?

Example, does beta have to decrease when lambda increase to ensure that X is the same?


1 Answer 1


The function you have described is a loss function. It is the function which we want to minimize in order to train our model. The loss function is called ordinary least squares. We can also see that the model that we are trying to fit is a linear function with model parameters $\beta$. This is an optimization function and regularizarion is often used in optimization problems to attain a solution which is less likely to be a result of overfitting.

No regularization

Without any regularization the solution to a linear regression is


L2 regularization

To understand regression it is much easier to first start with the more widely used L2 regularization, ridge regression. This is defined as

$\min_\beta (Y-X\beta)^T(Y-X\beta) + \lambda ||\beta||^2$

We can solve this to get a closed-form solution using a derivative

$0 = -2X^TY + 2X^TX\beta+2\lambda\beta$

$X^TY = (X^TX + \lambda I)\beta$

$\beta = (X^TX + \lambda I)^{-1}X^TY$

Let's analyze this result in contrast to the solution without regularization and see what it means. We can see that the only difference is the added regularization term $\lambda$ within the inverse. As we should know, the inverse of a larger value, will cause its result to get smaller. For example the inverse of 1/5, is larger than 1/2. So by adding the regularization term we actually reduced the associated weight $\beta$.

This is why regularizers are often called a penalty term. They further constrain the optimization function. By adding an additional cost associated to each weight, we can reduce the impact that each weight will have on our model. Thus reducing the potential for overfitting.

L1 regularization

L1 regularization is the penalty used for LASSO regression, it's cost function is defined as

$\min_\beta(Y-X\beta)^T(Y-X\beta) + \lambda||\beta||$

From what we learned above we can already tell that this additional cost will cause the resulting weights to be penalized. Unfortunately L1 regularization does not have a closed form solution because it is not differentiable when a weight $\beta$ falls to 0. Thus this requires some more work to solve. LASSO is an algorithm for finding the solution.

First let's remember that our constraint here is $||\beta||\leq c$ where c is a real number which is inversely related to $\lambda$. The plan is to find an appropriate $\lambda$ such that the minimum of the function falls within or on the contour of the constraint. This constraint has very sharp edges which lie on each dimensional axis at a distance $c$ from the origin. You can imagine it like a diamond in 2D space, an octohedron in 3D space, etc.

In high dimensional space these spikes have a very high likelyhood to being hit by the function you wish to optimize thus this will cause many of the features to have an associated weight of 0. For example if we compare the regularization of this line in 2D space using both L2 and L1 regularization.

enter image description here

In this picture the distance $r$ is equivalent to the factor $c$ I used above which is inversely related to the regularization term $\lambda$. The function being optimized (the cost function) is the described using the ellipses which represent their levels (like an altitude map). The shape at the center of the axis is the regularizer which is used to constrain the weights associated to each feature.

We can first notice that the original function is minimized exactly at its minimum. However, choosing this solution has a high probability of being one which overfits the training data.

We see that L2 regularization did add a penalty to the weights, we ended up with a constrained weight set. However both weights are still represented in your final solution. The function being optimized touches the surface of the regularizer in the first quadrant.

However, in the L1 case we can see that $\theta_2$ is the only feature that is required for this regression problem and $\theta_1$ will have a weight set to 0. Thus we can ignore it. It just happens that smooth functions encountering a function with pronounced edges tends to hit these edges with higher probability than hit the vertices. I do not know the proof for this, but I have seen it in a course as supplemental material and it was complicated. This is why L1 regularization is often used for feature selection.

Combining both together

What is often done is first using L1 regularization to find out what features have lasso weights which tend to 0, these are then removed from the original feature set. Then with this new feature set we can apply L2 regularization in order to still control these new features to control for overfitting.

  • $\begingroup$ Thanks, i understood the L2 part. For L1, what do you mean by spikes have high likelihood of being hit by your function and thus this will cause many of the features to have an associated weight of 0?Why does being hit by the function cuse the features weights to go to 0? $\endgroup$
    – jxn
    Commented Oct 13, 2018 at 9:33
  • $\begingroup$ I'll add to the answer so you can see what I mean by adding a constraint. $\endgroup$
    – JahKnows
    Commented Oct 13, 2018 at 15:33
  • $\begingroup$ @jxn, my pleasure, let me know if you'd like more details. $\endgroup$
    – JahKnows
    Commented Oct 14, 2018 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.