Difference between Ridge and Linear Regression

Question

From what I have understood, the Ridge Regression is just having the loss function for an optimization problem with the addition of the regularization term (L2 Norm in the case of Ridge). However I am not sure if the loss function can be described by a non-linear function or it needs to be linear. In this case, if the loss functions needs to be linear, then from what I understand the Ridge regression, is simply performing Linear regression with the addition of the L2-Norm for regularization. Please correct me if I am wrong.

Answer 1

Introduction to Statistical Learning (page261) gives some instructive details. The linear regression loss function is simply augmented by a penalty term in an additive way.

Answer 2

Yes, ridge regression is ordinary least squares regression with an L2 penalty term on the weights in the loss function. The loss function is not really linear in any of its terms, right? it's the squared residuals plus squares of weights. But there is no reason the loss function needs to be linear. It's helpful if it's convex, which it is here, but even that is not required to try to minimize it.

Answer 3

Linear Regression

It is one of the most widely known modeling technique. Linear regression is usually among the first few topics which people pick while learning predictive modeling. In this technique, the dependent variable is continuous, independent variable(s) can be continuous or discrete, and nature of regression line is linear.

Linear Regression establishes a relationship between dependent variable (Y) and one or more independent variables (X) using a best fit straight line (also known as regression line).

Ridge Regression

Ridge Regression is a technique used when the data suffers from multicollinearity ( independent variables are highly correlated). In multicollinearity, even though the least squares estimates (OLS) are unbiased, their variances are large which deviates the observed value far from the true value. By adding a degree of bias to the regression estimates, ridge regression reduces the standard errors.

Above, we saw the equation for linear regression. Remember? It can be represented as:

y=a+ b*x

This equation also has an error term. The complete equation becomes: y=a+b*x+e (error term), [error term is the value needed to correct for a prediction error between the observed and predicted value] => y=a+y= a+ b1x1+ b2x2+…+e, for multiple independent variables.

In a linear equation, prediction errors can be decomposed into two sub components. First is due to the biased and second is due to the variance. Prediction error can occur due to any one of these two or both components. Here, we’ll discuss about the error caused due to variance.

Ridge regression solves the multicollinearity problem through shrinkage parameter λ (lambda). Look at the equation below.

Ridge

In this equation, we have two components. First one is least square term and other one is lambda of the summation of β2 (beta- square) where β is the coefficient. This is added to least square term in order to shrink the parameter to have a very low variance.

Important Points: •The assumptions of this regression is same as least squared regression except normality is not to be assumed •It shrinks the value of coefficients but doesn’t reaches zero, which suggests no feature selection feature •This is a regularization method and uses l2 regularization.