(I asked this in mathematics site, but nobody responded, it seems the whole problem is more related to data science than math.)

In a regression problem, loss function is:

$$L(a,b) = {\sum_{i=1}^n (y^i - (ax^i +b))^2})$$

In order to minimize this value, we need to set the derivative of L with respect to each of its parameters, equal to zero.

Hence, $\frac{dL}{db}$ would be $y^- + a \cdot x^-$ But what would $\frac{dL}{da}$ be?

$$\frac{\sigma L(a,b)}{\sigma a} = {2 \cdot \sum_{i=1}^n (y^i - (ax^i +b)}) \cdot \frac{\sigma \sum(y^i - (ax^i +b))}{\sigma a}$$ $$\frac{\sigma L(a,b))}{\sigma a} = {2 \cdot \sum_{i=1}^n (y^i - (ax^i +b)}) \cdot -\sum(x^i)$$ $$\frac{\sigma L(a,b))}{\sigma a} = {2\sum(x^i) \cdot \sum_{i=1}^n (y^i) - 2\sum(x^i) \cdot \sum_{i=1}^n (ax^i) - 2\sum(x^i) \cdot \sum_{i=1}^nb}$$ $$\frac{\sigma L(a,b))}{\sigma a} = {2\sum_{i=1}^n (y^i \cdot x^i) - 2\sum_{i=1}^n (ax^i \cdot x^i) - 2\sum_{i=1}^n x^i \cdot b}$$ $$\frac{\sigma L(a,b))}{\sigma a} = {2\sum_{i=1}^n (y^i \cdot x^i) - 2\sum_{i=1}^n ((ax^i +b) \cdot x^i)} = 0$$

How this would be equal to $\frac{cov(x, y)}{\sigma^2x}$


For a linear regression we have the loss function


The partial derivatives are

$$\dfrac{\partial J}{\partial a}=2\sum_{n=1}^N(y_n-a-bx_n)(-1)$$ $$\dfrac{\partial J}{\partial b}=2\sum_{n=1}^N(y_n-a-bx_n)(-x_n).$$

If we set both derivatives to zero and divide by the sample size $N$ we obtain

$$0=\overline{y}-a-b\overline{x}$$ $$0=\overline{xy}-a\overline{x}-b\overline{x^2}.$$

Now, solve the first equation for $a= \overline{y}-b\overline{x}$ and plug this into the second equation


and solve for $b$ to obtain


The espression in the numerator is the covariance for a sample and the expression in the denominator is the variance of $x$.

  • $\begingroup$ Thank you so much. It was really helpful. $\endgroup$ Jul 14 '19 at 0:50
  • 1
    $\begingroup$ You are welcome :). $\endgroup$ Jul 14 '19 at 7:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.