0
$\begingroup$

Linear regression is used when there is a linear relationship between the input and output variables. Does this linear relationship mean that there is no power over the variables or the parameters? In m understanding, by linear we mean linear with respect to the parameters (no power). Please correct me if wrong. Basically, I am a bit confused looking at this tutorial: https://www.gaussianwaves.com/2013/03/how-to-estimate-unknown-parameters-using-ordinary-least-squares-ols/

where linear regression is used to fit the straight line but if the function is a cubic or higher order polynomial (power over the variables), can we still use linear regression? I have seen that for fitting higher order polynomials, we use Regularized Linear Regression in order to penalize the higher order termed variables. Based on these confusions, can somebody please help clarify the following points?

MAIN QUESTION FOR WHICH BOUNTY IS OPENED:

1) Linearity is with respect to the variables or the parameters? I saw a similar question asked here Is a "curve" considered "linear"? but the second answer says that linearity is in terms of the parameters such as weights or other hyperparameters and polynomial regression is a special type of linear regression. So, if the parameters of the model have a power then that function is a nonlinear function and hence we get a nonlinear model. Is that correct?


Other questions that have been answered and understood by me

2) Do higher order termed polynomials fall under linear or non-linear models?

3)The sigmoid function does not have any power and is commonly used in the hidden layers of multi layer perceptron thereby making MLP non-linear machine learning model? How come sigmoid curve is a non-linear function?

4) Is there a check whether to go for linear or non-linear functions ie., whether to fit linear regression or non-linear models such as SVM with kernel?

$\endgroup$
6
+50
$\begingroup$

I think it is helpful to distinguish between linear functions (representing the relationship between independent and dependent variables) and linear models (representing the relationship between the model parameters and the outcome).

A linear model can be represented by a non-linear function. A linear regression model is any model that is represented by a linear function in the parameters. A polynomial can be represented as such.

Likewise a nonlinear regression model is any model that is represented by a nonlinear combination of parameters.

To quote from Wikipedia:

Although polynomial regression fits a nonlinear model to the data, as a statistical estimation problem it is linear, in the sense that the regression function E(y | x) is linear in the unknown parameters that are estimated from the data.

Let the model be represented by the function $f(x,w)$ with inputs $x$ and parameter vector $w$. Polynomial regression has the form $f(x,w) = w_0 + w_1*x + w_2*x^2 \ldots$ It is linear in the parameters because $f(x,a+b) = (a_0+b_0) + (a_1+b_1)*x + (a_2+b_2)*x^2 \ldots = a_0 + a_1*x + a_2*x^2 \ldots + b_0 + b_1*x + b_2*x^2 \ldots = f(x,a) + f(x,b)$. But it is not linear in the inputs because $f(x+y,w) \neq f(x,w)+f(y,w)$.

Often people think about the relationship $y=f(x)$ between $x$ and $y$, but linear in linear regression refers to linearity in $w$.

$\endgroup$
  • $\begingroup$ Thank you for your answer. These points are quite unclear:(1) Can you please elaborate a bit more on linear models with some examples ( I think multi layer perceptron is a nonlinear model but it also uses a sigmoid which is a nonlinear function; thus is a model nonlinear because it uses a nonlinear function)? (2) Is a quadratic function having powers over the variable a nonlinear function and hence we can get a nonlinear regression? $\endgroup$ – Sm1 Oct 31 at 18:09
  • $\begingroup$ It's really that simple... linearity needs only to be in the parameters (1) a multilayer perceptron with sigmoid activation is nonlinear in both the parameters and the inputs (2) it's a nonlinear function but a linear regression model because it is still linear in the parameters... see updated answer $\endgroup$ – oW_ Oct 31 at 19:02
  • $\begingroup$ good answer, upvote $\endgroup$ – Peter Oct 31 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.