0
$\begingroup$

(I asked this in mathematics site, but nobody responded, it seems the whole problem is more related to data science than math.)

In a regression problem, loss function is:

$$L(a,b) = {\sum_{i=1}^n (y^i - (ax^i +b))^2})$$

In order to minimize this value, we need to set the derivative of L with respect to each of its parameters, equal to zero.

Hence, $\frac{dL}{db}$ would be $y^- + a \cdot x^-$ But what would $\frac{dL}{da}$ be?

$$\frac{\sigma L(a,b)}{\sigma a} = {2 \cdot \sum_{i=1}^n (y^i - (ax^i +b)}) \cdot \frac{\sigma \sum(y^i - (ax^i +b))}{\sigma a}$$ $$\frac{\sigma L(a,b))}{\sigma a} = {2 \cdot \sum_{i=1}^n (y^i - (ax^i +b)}) \cdot -\sum(x^i)$$ $$\frac{\sigma L(a,b))}{\sigma a} = {2\sum(x^i) \cdot \sum_{i=1}^n (y^i) - 2\sum(x^i) \cdot \sum_{i=1}^n (ax^i) - 2\sum(x^i) \cdot \sum_{i=1}^nb}$$ $$\frac{\sigma L(a,b))}{\sigma a} = {2\sum_{i=1}^n (y^i \cdot x^i) - 2\sum_{i=1}^n (ax^i \cdot x^i) - 2\sum_{i=1}^n x^i \cdot b}$$ $$\frac{\sigma L(a,b))}{\sigma a} = {2\sum_{i=1}^n (y^i \cdot x^i) - 2\sum_{i=1}^n ((ax^i +b) \cdot x^i)} = 0$$

How this would be equal to $\frac{cov(x, y)}{\sigma^2x}$

$\endgroup$
3
$\begingroup$

For a linear regression we have the loss function

$$J(a,b)=\sum_{n=1}^N(y_n-a-bx_n)^2.$$

The partial derivatives are

$$\dfrac{\partial J}{\partial a}=2\sum_{n=1}^N(y_n-a-bx_n)(-1)$$ $$\dfrac{\partial J}{\partial b}=2\sum_{n=1}^N(y_n-a-bx_n)(-x_n).$$

If we set both derivatives to zero and divide by the sample size $N$ we obtain

$$0=\overline{y}-a-b\overline{x}$$ $$0=\overline{xy}-a\overline{x}-b\overline{x^2}.$$

Now, solve the first equation for $a= \overline{y}-b\overline{x}$ and plug this into the second equation

$$0=\overline{xy}-\overline{x}\overline{y}+b\overline{x}^2-b\overline{x^2}$$

and solve for $b$ to obtain

$$b=\dfrac{\overline{xy}-\overline{x}\overline{y}}{\overline{x^2}-\overline{x}^2}.$$

The espression in the numerator is the covariance for a sample and the expression in the denominator is the variance of $x$.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you so much. It was really helpful. $\endgroup$ – Fatemeh Asgarinejad Jul 14 '19 at 0:50
  • 1
    $\begingroup$ You are welcome :). $\endgroup$ – MachineLearner Jul 14 '19 at 7:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.