The model of linear regression is linear in parameters.
What does this actually mean?
$\begingroup$
$\endgroup$
-
1$\begingroup$ More reference, if possible? Something above or below that line (from the text you found this)? $\endgroup$ – Dawny33♦ Jun 19 '16 at 5:46
-
$\begingroup$ This looks like, and its hard to tell from the lack of context, a stats question, and should probably be in an expanded form on the stats stackexchange site. $\endgroup$ – Spacedman Jun 19 '16 at 17:31
$\begingroup$
$\endgroup$
Consider an equation of the form
$y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \epsilon$
where $x$'s are the variables and $\beta$'s are the parameters. Here, y is a linear function of $\beta$'s (linear in parameters) and also a linear function of $x$'s (linear in variables). If you change the equation to
$y = \beta_0 + \beta_1x_1 + \beta_2x_1^2 + \epsilon$
Then, it is no longer linear in variables (because of the squared term) but it is still linear in parameters. And for (multiple) linear regression, that's all that matters because in the end, you are trying to find a set of $\beta$'s that minimizes a loss function. For that, you need to solve a system of linear equations. Given its nice properties, it has a closed form solution that makes our lives easier. Things get harder when you deal with nonlinear equations.
Assume you are not dealing with a regression model but instead you have a mathematical programming problem: You are trying to minimize an objective function of the form $c^Tx$ subject to a set of constraints: $Ax \geq b$ and $x\geq0$. This is a linear programming problem in the sense that it is linear in variables. Unlike the regression model, you are trying to find a set of $x$'s (variables) that satisfies the constraints and minimizes the objective function. This will also require you to solve systems of linear equations but here it will be linear in variables. Your parameters won't have any effect on that system of linear equations.
$\begingroup$
$\endgroup$
It simply means that $Y = A X$ where $A$ are the parameters. The variables $X$ might contain nonlinear relationships; e.g., $X = [\alpha\; \alpha \beta\; \beta^2]^T$, yet $Y$ is a linear function of $X$.
-
1$\begingroup$ Thanks! In your context, Doesn't it regular consider X as the variable and A as the parameter? $\endgroup$ – Albert Gao Jun 19 '16 at 7:35
-
2$\begingroup$ Are you sure X is the parameters? I would say the matrix A is the parameters . . . $\endgroup$ – Neil Slater Jun 19 '16 at 7:37
-
$\begingroup$ Surely $X$ are the variables here and $A$ the parameters which may or may not be "linear in the parameters". They are only linear if the assume a non-linear form. The model is only linear if that is true and the variables also only have a linear form. $\endgroup$ – Astrid Dec 1 '17 at 6:05
$\begingroup$
$\endgroup$
A model is linear when each term is either a constant or the product of a parameter and a predictor. A linear equation is constructed by adding the results for each term. This constrains the equation to just one basic form:
$Response = constant + parameter * predictor + ... + parameter * predictor$
"Linear in parameters" in Linear Regression, means no parameter appears as an exponent, nor multiplied or divided by another parameter.
