A linear regression line has an equation of the form $y = \beta_0 + \beta_1 x$,
where $x$ is the explanatory variable and $y$ is the dependent
variable. The slope of the line is $\beta_1$ (coeff), and $\beta_0$ is the
intercept (the value of $y$ when $x = 0$).
The most common method for fitting a regression line is the method of
least-squares. This method calculates the best-fitting line for the
observed data by minimizing the sum of the squares of the vertical
deviations from each data point to the line (if a point lies on the
fitted line exactly, then its vertical deviation is 0). Because the
deviations are first squared, then summed, there are no cancellations
between positive and negative values.
Ordinary least squares (OLS) is the most common estimator. OLS
estimates are commonly used to analyze both experimental and
observational data. The OLS method minimizes the sum of squared
residuals, and leads to a closed-form expression for the estimated
value of the unknown parameter vector $β$:
$\hat {\boldsymbol {\beta }}=(\mathbf {X} ^{\mathsf {T}}\mathbf {X} )^{-1}\mathbf {X} ^{\mathsf {T}}\mathbf {Y}$ where $Y$ is a vector
whose ith element is the ith observation of the dependent variable,
and $X$ is a matrix whose ij element is the ith observation of the jth
independent variable.
NOTE Higher-order polynomial regression can be treated as an extension to linear regression by treating $x^2$, $x^3$, .. as extra features and using linear least squares with previous formula. That is one uses linear least squares on $y = \beta_0 + \beta_1 x + \beta_2 x^2 + ..$ where $x$, $x^2$, $x^3$, .. are assumed as different features.
In statistics, polynomial regression is a form of regression analysis
in which the relationship between the independent variable x and the
dependent variable y is modelled as an nth degree polynomial in x.
Polynomial regression fits a nonlinear relationship between the value
of x and the corresponding conditional mean of y, denoted E(y |x).
Although polynomial regression fits a nonlinear model to the data, as
a statistical estimation problem it is linear, in the sense that the
regression function E(y | x) is linear in the unknown parameters that
are estimated from the data. For this reason, polynomial regression is
considered to be a special case of multiple linear regression.
References:
- http://www.stat.yale.edu/Courses/1997-98/101/linreg.htm
- https://en.wikipedia.org/wiki/Linear_least_squares
- https://en.wikipedia.org/wiki/Polynomial_regression
