# Difference between Gradient Descent and Normal Equation in Linear Regression

Hi I am new to Linear Regression. I want to know

what is the difference b/w Gradient Descent and Mean Square Error in Linear Regression using machine learning?

And

When to use Gradient Descent and Mean Square Error in Linear Regression using machine learning? Or

When to use which algorithm in Linear Regression.?

Can anyone explain.?

• Mean Square Error is a loss function not an algorithm. You don't need to choose between MSE and Gradient Descent. Did you perhaps mean to name a different algorithm (e.g. the Normal Equation)? – Neil Slater Oct 4 '18 at 7:56

## 2 Answers

To train a model, two processes have to be followed. From the predicted output, the error has to be calculated w.r.t the real output. Once the error is calculated, the weights of the model has to be changed accordingly.

Mean square error is a way of calculating the error. Depending upon the type of output, the error calculation differs. There are absolute errors, cross-entropy errors, etc. The cost function and error function are almost the same.

Gradient descent is an optimization algorithm or simply update rule, used to change the weight values. Some of the variations are Stochastic gradient descent, momentum, AdaGrad, AdaDelta, RMSprop, etc. More about Optimization algorithms

• :- Hi Can you please tell me when to use Gradient Descent in Linear Regression..? Because their might be different algorithm .? Why only Gradient Descent.? Is their any specific reason for using Gradient Descent.? – Sanjiv Oct 4 '18 at 11:11
• Go through the link provided. Optimization algorithms are mostly about quickly reaching the minimum point. There are multiple ways to reach it. There is no strong rule which is the best one. Basically gradient descent is obtaining the slope of the cost function and going in the negative direction, so as to minimize the cost. – chmodsss Oct 4 '18 at 11:53

Gradient Descent is an algorithm for minimizing some functions like the Mean Squared Error function. As described, GD takes iterative steps downhill in the negative gradient direction to find the min. Normal Equations are a way of directly solving the quadratic eqn for the weights that minimize the MSE. But solving Normal Eqns involves inverting a matrix which is of size N_features x N_features, an O(N_features**3) operation. So when N_features > 1000 or so it starts to slow while GD is O(N).