Why does reducing polynomial regression to linear regression work?

Question

Getting into machine learning, have a reasonable background in statistics and understand the basic principles of linear algebra (matrix multiplication etc.) - but am having a damn hard time figuring out why reducing a polynomial regression works.

For example, say we have this function:

$y =$ $\beta_0$ + $\beta_1$$x_1$ + $\beta_2$$x_2$$^2$

From what I've seen on 4 videos and 6 articles, we can use the following substitutions:

• $x_2$ = 1
• $x_3$ = $x$
• $x_4$ = $x^2$

To create the following model: $y =$ $\beta_0$ + $\beta_1$$x_1$ + ($\beta_2$$x_2$ + $\beta_3$$x_3$ + $\beta_4$$x_4$)

And then, fine - we can solve that as a normal multiple linear regression, and all is well.

But why, why does this work? I really cannot find an explanation for this.

Thanks!

Answer 1

I don't really understand your problem statement. You can simply add squared terms to any linear model. Say you have $y$ and $x$ and you want to model a polynomial function, you can write a model like:

$y=\beta_0 + \beta_1 x_1 + \beta_2 x_1^2 + u$.

In matrix form this would look like

$y=X\beta + u$.

An example with some numbers would write:

$\left( \begin{array}{rrrr} 5 \\ 8 \\ 7 \\ 6 \\ \end{array}\right) $ = $\left( \begin{array}{rrrr} 1 & 8& 8^2 \\ 1 & 4 & 4^2 \\ 1 & 3 & 3^2 \\ 1 & 6 & 6^2 \\ \end{array}\right) \beta + u.$

You can solve this like $\hat{\beta} = (X'X)^{-1} X'y$, where $\hat{\beta}$ are the coefficients of the linear regression model.