Linear regression is used when there is a linear relationship between the input and output variables. Does this linear relationship mean that there is no power over the variables or the parameters? In m understanding, by linear we mean linear with respect to the parameters (no power). Please correct me if wrong. Basically, I am a bit confused looking at this tutorial: https://www.gaussianwaves.com/2013/03/how-to-estimate-unknown-parameters-using-ordinary-least-squares-ols/
where linear regression is used to fit the straight line but if the function is a cubic or higher order polynomial (power over the variables), can we still use linear regression? I have seen that for fitting higher order polynomials, we use Regularized Linear Regression in order to penalize the higher order termed variables. Based on these confusions, can somebody please help clarify the following points?
MAIN QUESTION FOR WHICH BOUNTY IS OPENED:
1) Linearity is with respect to the variables or the parameters? I saw a similar question asked here Is a "curve" considered "linear"? but the second answer says that linearity is in terms of the parameters such as weights or other hyperparameters and polynomial regression is a special type of linear regression. So, if the parameters of the model have a power then that function is a nonlinear function and hence we get a nonlinear model. Is that correct?
Other questions that have been answered and understood by me
2) Do higher order termed polynomials fall under linear or non-linear models?
3)The sigmoid function does not have any power and is commonly used in the hidden layers of multi layer perceptron thereby making MLP non-linear machine learning model? How come sigmoid curve is a non-linear function?
4) Is there a check whether to go for linear or non-linear functions ie., whether to fit linear regression or non-linear models such as SVM with kernel?