I am new to machine learning and am a bit confused about the definition of a linear model. I've searched many sources and the most common definition is:

The term linear model implies that the model is specified as a linear combination of features.

Source: https://docs.aws.amazon.com/machine-learning/latest/dg/linear-models.html

As I understood, when we speak about linear classifier, we mean an algorithm like that: $a(x)=w_1 * x_1 + w_2 * x_2 + ... + w_n * x_n$, where $w_i$ - weights, $x_i$ - features. So, the question is, term "linear" means function, which is mathematically linear by feature or by weight? For example, in the task, where we have only one feature, can we say, that algorithm $a(x) = w*((x)^2)$ is linear classifier? The same question about $a(x)=((w)^2)*(x)$?



2 Answers 2


It's linear by Feature. In the end, we are modelling the Features.
So, A linear model is the one that plots a linear function in the Feature-Label space.
e.g. LinearRegression, Linear SVM, LogisticRegression.

Confusion arises when we try to look at the True relation but we should look at the modelled relation.

Let's say we have a True relation $y = w_r*x^2$ and we don't know this.
The feature I got is $x$, so the model that I will build is $y = w_p*x$ and it is a linear model with the given feature.
Definitely, by doing EDA I will realize that the relationship is violating the Linear assumption. So, I will create a new Feature i.e. $x^2$
Let's assume, $x^2 = F$. Its a new space now i.e. $x {\rightarrow} y$ to $x^2 {\rightarrow} y$
My new model will be $y = w_r*F$ and again it is a Linear relationship.

On weights,
Weights are the parameters to learn i.e. slope for a LinearRegression. It represents the significance of each Features.

On polynomial,
Although we use polynomial plot to develop intuition but we use polynomial features(as explained above) to fit the model. So, it is still a Linear model.

$\hspace{6cm}$Excerpt from ISL(Page - 91) enter image description here

Short answer

Linear means linear "by weight" (as you express it). Yes, $a(x) = w*((x)^2)$ is linear.

Longer answer

In your second example $a(x)=((w)^2)*(x)$, the weight $w$ is initially unknown, the model learns it. Therefore, the expression $(w)^2$ should just be renamed as $(w_1)$ and your model is linear

The example of the nonlinear model would be $a(x) = w_1*x^{(w_2)}$, where $w_1$ and $w_2$ are weights to be learned.

  • $\begingroup$ Isn't the relation between $x^2$ and $x$ non-linear by definition? I think I disagree with your answer. While you can define a new feature $x'=x^2$ and then fit a linear model on the new feature, if the feature is $x$ then $w x^2$ is a polynomial model. $\endgroup$
    – noe
    Jun 3, 2021 at 19:20
  • $\begingroup$ Thanks for answers! Actually, I am closer to @noe opinion, but I am not sure. Can someone else comment on my question to gain the truth? $\endgroup$
    – AlexM
    Jun 4, 2021 at 5:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.