# Coefficients of Linear regression for minimizing MSE

(I asked this in mathematics site, but nobody responded, it seems the whole problem is more related to data science than math.)

In a regression problem, loss function is:

$$L(a,b) = {\sum_{i=1}^n (y^i - (ax^i +b))^2})$$

In order to minimize this value, we need to set the derivative of L with respect to each of its parameters, equal to zero.

Hence, $$\frac{dL}{db}$$ would be $$y^- + a \cdot x^-$$ But what would $$\frac{dL}{da}$$ be?

$$\frac{\sigma L(a,b)}{\sigma a} = {2 \cdot \sum_{i=1}^n (y^i - (ax^i +b)}) \cdot \frac{\sigma \sum(y^i - (ax^i +b))}{\sigma a}$$ $$\frac{\sigma L(a,b))}{\sigma a} = {2 \cdot \sum_{i=1}^n (y^i - (ax^i +b)}) \cdot -\sum(x^i)$$ $$\frac{\sigma L(a,b))}{\sigma a} = {2\sum(x^i) \cdot \sum_{i=1}^n (y^i) - 2\sum(x^i) \cdot \sum_{i=1}^n (ax^i) - 2\sum(x^i) \cdot \sum_{i=1}^nb}$$ $$\frac{\sigma L(a,b))}{\sigma a} = {2\sum_{i=1}^n (y^i \cdot x^i) - 2\sum_{i=1}^n (ax^i \cdot x^i) - 2\sum_{i=1}^n x^i \cdot b}$$ $$\frac{\sigma L(a,b))}{\sigma a} = {2\sum_{i=1}^n (y^i \cdot x^i) - 2\sum_{i=1}^n ((ax^i +b) \cdot x^i)} = 0$$

How this would be equal to $$\frac{cov(x, y)}{\sigma^2x}$$

For a linear regression we have the loss function

$$J(a,b)=\sum_{n=1}^N(y_n-a-bx_n)^2.$$

The partial derivatives are

$$\dfrac{\partial J}{\partial a}=2\sum_{n=1}^N(y_n-a-bx_n)(-1)$$ $$\dfrac{\partial J}{\partial b}=2\sum_{n=1}^N(y_n-a-bx_n)(-x_n).$$

If we set both derivatives to zero and divide by the sample size $$N$$ we obtain

$$0=\overline{y}-a-b\overline{x}$$ $$0=\overline{xy}-a\overline{x}-b\overline{x^2}.$$

Now, solve the first equation for $$a= \overline{y}-b\overline{x}$$ and plug this into the second equation

$$0=\overline{xy}-\overline{x}\overline{y}+b\overline{x}^2-b\overline{x^2}$$

and solve for $$b$$ to obtain

$$b=\dfrac{\overline{xy}-\overline{x}\overline{y}}{\overline{x^2}-\overline{x}^2}.$$

The espression in the numerator is the covariance for a sample and the expression in the denominator is the variance of $$x$$.

• Thank you so much. It was really helpful. – Fatemeh Asgarinejad Jul 14 '19 at 0:50
• You are welcome :). – MachineLearner Jul 14 '19 at 7:31