0
$\begingroup$

I am new to machine learning.

I have read about the linear regression where-in the ideal model is a line which has the least mean squared error. In multi-variable linear regression we would have a plane instead of the line. However the goal is the same - least mean squared error. This is calculated as the sum of squared distance between the value and the line, divided by the number of samples.

I then read about gradient descent which is a technique to reduce the loss. So the example showed a loss function with y-axis as loss and x-axis as model params.

I'm trying to understand the relation between gradient descent and linear regression's least mean squared error.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Related // What is your understanding about a loss function vs a regression function? $\endgroup$
    – Dave
    Jul 29 at 9:57
  • $\begingroup$ I'm new to this topic and when learning about linear regression they showed mean squared error as way to minimize loss. Where as the next course section said about gradient descent. So I'm not sure whether there is a relation between MSE and GD? $\endgroup$
    – variable
    Jul 29 at 10:06
  • $\begingroup$ Mean squared error is the loss. Then you need some way to figure out the regression parameters that minimize the loss. In linear regression, we are lucky enough to have calculus give a simple equation involving matrix inversions and transposes, $\hat\beta_{OLS}=(X^TX)^{-1}X^Ty$, but other methods like logistic regression and neural networks are not so lucky and have to rely on numerical approximation like gradient descent. // If we have an equation like we do in OLS and choose to do gradient descent, gradient descent should give an answer close to that given by the explicit equation. $\endgroup$
    – Dave
    Jul 29 at 10:16

1 Answer 1

Reset to default
2
$\begingroup$

As Dave has mentioned, linear regression is simple enough to allow an analytical solution from the cost function. To determine this formula, you can start from the cost function formula. Then, by computing the gradient (meaning the derivative of J with respect to any theta coefficients) and search when this gradient is equal to 0, you find that the solution is :

linear regression solution

An other advantage of gradient descent is that is faster than the analytical solution when X is very large. If you take a look on the solution formula, you can see it asks for a lot of computing ressources when when inverting the matrix.

linear regression complexity

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.