The model of linear regression is linear in parameters.
What does this actually mean?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this communityThe model of linear regression is linear in parameters.
What does this actually mean?
Consider an equation of the form
$y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \epsilon$
where $x$'s are the variables and $\beta$'s are the parameters. Here, y is a linear function of $\beta$'s (linear in parameters) and also a linear function of $x$'s (linear in variables). If you change the equation to
$y = \beta_0 + \beta_1x_1 + \beta_2x_1^2 + \epsilon$
Then, it is no longer linear in variables (because of the squared term) but it is still linear in parameters. And for (multiple) linear regression, that's all that matters because in the end, you are trying to find a set of $\beta$'s that minimizes a loss function. For that, you need to solve a system of linear equations. Given its nice properties, it has a closed form solution that makes our lives easier. Things get harder when you deal with nonlinear equations.
Assume you are not dealing with a regression model but instead you have a mathematical programming problem: You are trying to minimize an objective function of the form $c^Tx$ subject to a set of constraints: $Ax \geq b$ and $x\geq0$. This is a linear programming problem in the sense that it is linear in variables. Unlike the regression model, you are trying to find a set of $x$'s (variables) that satisfies the constraints and minimizes the objective function. This will also require you to solve systems of linear equations but here it will be linear in variables. Your parameters won't have any effect on that system of linear equations.
It simply means that $Y = A X$ where $A$ are the parameters. The variables $X$ might contain nonlinear relationships; e.g., $X = [\alpha\; \alpha \beta\; \beta^2]^T$, yet $Y$ is a linear function of $X$.
A model is linear when each term is either a constant or the product of a parameter and a predictor. A linear equation is constructed by adding the results for each term. This constrains the equation to just one basic form:
$Response = constant + parameter * predictor + ... + parameter * predictor$
"Linear in parameters" in Linear Regression, means no parameter appears as an exponent, nor multiplied or divided by another parameter.
To add to this. The formal definition of linearity of a function f is: $f(a*z1+b*z2)=a*f(z1)+b*f(z2)$