Definition of linear model

Question

I am new to machine learning and am a bit confused about the definition of a linear model. I've searched many sources and the most common definition is:

The term linear model implies that the model is specified as a linear combination of features.

Source: https://docs.aws.amazon.com/machine-learning/latest/dg/linear-models.html

As I understood, when we speak about linear classifier, we mean an algorithm like that: $a(x)=w_1 * x_1 + w_2 * x_2 + ... + w_n * x_n$, where $w_i$ - weights, $x_i$ - features. So, the question is, term "linear" means function, which is mathematically linear by feature or by weight? For example, in the task, where we have only one feature, can we say, that algorithm $a(x) = w*((x)^2)$ is linear classifier? The same question about $a(x)=((w)^2)*(x)$?

Thanks!

Answer 1

It's linear by Feature. In the end, we are modelling the Features.
So, A linear model is the one that plots a linear function in the Feature-Label space.
e.g. LinearRegression, Linear SVM, LogisticRegression.

Confusion arises when we try to look at the True relation but we should look at the modelled relation.

Let's say we have a True relation $y = w_r*x^2$ and we don't know this.
The feature I got is $x$, so the model that I will build is $y = w_p*x$ and it is a linear model with the given feature.
Definitely, by doing EDA I will realize that the relationship is violating the Linear assumption. So, I will create a new Feature i.e. $x^2$
Let's assume, $x^2 = F$. Its a new space now i.e. $x {\rightarrow} y$ to $x^2 {\rightarrow} y$
My new model will be $y = w_r*F$ and again it is a Linear relationship.

On weights,
Weights are the parameters to learn i.e. slope for a LinearRegression. It represents the significance of each Features.

On polynomial,
Although we use polynomial plot to develop intuition but we use polynomial features(as explained above) to fit the model. So, it is still a Linear model.

$\hspace{6cm}$^{Excerpt from ISL(Page - 91)}

Answer 2

Short answer

Linear means linear "by weight" (as you express it). Yes, $a(x) = w*((x)^2)$ is linear.

Longer answer

In your second example $a(x)=((w)^2)*(x)$, the weight $w$ is initially unknown, the model learns it. Therefore, the expression $(w)^2$ should just be renamed as $(w_1)$ and your model is linear

The example of the nonlinear model would be $a(x) = w_1*x^{(w_2)}$, where $w_1$ and $w_2$ are weights to be learned.