Learning cost function for linear regression

Question

(Me: Never learned calculus or advanced math and I started Stanford openclasses for machine learning. I know basic matrix calculations.)

One chapter of my course is about cost function. I have been trying to find any example calculation of it with numbers. Googling only finds the same formula everytime, and also on Octave. But I want to do the same thing first with pen+paper and without it, I cannot understand. Please give me a very simple example of using the formula with numbers. Thanks a lot.

I require a cost function calculation example for following sample dataset:

#Rooms = Rent
1 = 4000
2 = 10000
3 = 22000
4 = 30000

Answer 1

There is a worked out example in the Wikipedia page for "simple linear regression"

Just for the sake of it, let me plug in your example into the formulas:

The fitted model should be a straight line with parameters $\alpha$ (value at $x = 0$) and $\beta$ (the slope):

$$f(x) = \alpha + \beta x$$

The values for these parameters that minimize the distance between line and data points are called $\hat{\alpha}$ and $\hat{\beta}$. They can be computed out of the data point values by using these formulae, derived here:

$$\begin{align} \hat{\beta} & = \frac{ \overline{xy} - \bar{x}\bar{y} }{ \overline{x^2} - \bar{x}^2 } , \\ \\ \hat{\alpha} & = \bar{y} - \hat{\beta}\bar{x} \end{align} $$

where an expression with an overline $\overline{xy}$ means the sample average of that expression: $\overline{xy} = \tfrac{1}{n} \sum_{i=1}^n{x_iy_i}$.

Here are the values I find for the datapoints you have listed in your question:

$$ \begin{align} \overline{xy} &= \frac{1}{4} \sum{<(1 \times 4000), (2 \times 10000), (3 \times 22000), (4 \times 30000)>} \\ &= \frac{1}{4}(4000 + 20000 + 66000 + 120000) = 52500, \\ \overline{x} &= \frac{1}{4} \sum{<1, 2, 3, 4>} = 2.5 ,\\ \overline{y} &= \frac{1}{4} \sum{<4000, 10000, 22000, 30000>} = 16500 , \\ \overline{x^2} &= \frac{1}{4} \sum{<1^2, 2^2, 3^2, 4^2>} = 7.5 , \\ \overline{x}^2 &= 2.5^2 = 6.25 \end{align} $$

and the fitted line should be:

$$ \begin{align}\ \hat{\beta} &= \frac{52500 - 2.5 \times 16500}{7.5 - 6.25} = \frac{41250}{1.25} = 33000 , \\ \hat{\alpha} &= 16500 - 33000 \times 2.5 = - 66000 , \\ \Rightarrow f(x) &= - 66000 + 33000 x \end{align} $$

Therefore, the model would predict, for a house with 10 rooms, a rent of:

$$ f(x) = -66000 + 33000 \times 10 = 264000 $$