Regression: What defines Linear and non-linear models or functions

Question

Linear regression is used when there is a linear relationship between the input and output variables. Does this linear relationship mean that there is no power over the variables or the parameters? In m understanding, by linear we mean linear with respect to the parameters (no power). Please correct me if wrong. Basically, I am a bit confused looking at this tutorial: https://www.gaussianwaves.com/2013/03/how-to-estimate-unknown-parameters-using-ordinary-least-squares-ols/

where linear regression is used to fit the straight line but if the function is a cubic or higher order polynomial (power over the variables), can we still use linear regression? I have seen that for fitting higher order polynomials, we use Regularized Linear Regression in order to penalize the higher order termed variables. Based on these confusions, can somebody please help clarify the following points?

MAIN QUESTION FOR WHICH BOUNTY IS OPENED:

1) Linearity is with respect to the variables or the parameters? I saw a similar question asked here Is a "curve" considered "linear"? but the second answer says that linearity is in terms of the parameters such as weights or other hyperparameters and polynomial regression is a special type of linear regression. So, if the parameters of the model have a power then that function is a nonlinear function and hence we get a nonlinear model. Is that correct?

---

Other questions that have been answered and understood by me

2) Do higher order termed polynomials fall under linear or non-linear models?

3)The sigmoid function does not have any power and is commonly used in the hidden layers of multi layer perceptron thereby making MLP non-linear machine learning model? How come sigmoid curve is a non-linear function?

4) Is there a check whether to go for linear or non-linear functions ie., whether to fit linear regression or non-linear models such as SVM with kernel?

Answer 1

I think it is helpful to distinguish between linear functions (representing the relationship between independent and dependent variables) and linear models (representing the relationship between the model parameters and the outcome).

A linear model can be represented by a non-linear function. A linear regression model is any model that is represented by a linear function in the parameters. A polynomial can be represented as such.

Likewise a nonlinear regression model is any model that is represented by a nonlinear combination of parameters.

To quote from Wikipedia:

Although polynomial regression fits a nonlinear model to the data, as a statistical estimation problem it is linear, in the sense that the regression function E(y | x) is linear in the unknown parameters that are estimated from the data.

Let the model be represented by the function $f(x,w)$ with inputs $x$ and parameter vector $w$. Polynomial regression has the form $f(x,w) = w_0 + w_1*x + w_2*x^2 \ldots$ It is linear in the parameters because $f(x,a+b) = (a_0+b_0) + (a_1+b_1)*x + (a_2+b_2)*x^2 \ldots = a_0 + a_1*x + a_2*x^2 \ldots + b_0 + b_1*x + b_2*x^2 \ldots = f(x,a) + f(x,b)$. But it is not linear in the inputs because $f(x+y,w) \neq f(x,w)+f(y,w)$.

Often people think about the relationship $y=f(x)$ between $x$ and $y$, but linear in linear regression refers to linearity in $w$.