2
$\begingroup$

I'm new in Machine Learning and of the first concept I would like to learn is linear regression. I read that to apply linear regression I need to use a linear model. Starting from this assumption I know that this is a simple model for linear regression :

y = w0 + w1x

The definition of linear regression says that the dependent variable y should be a linear combination of the parameters w (but it is not necessary the same for the independent variable x )

So we can say that also this is a linear regression model :

y = w0 + w1x1 + w2(x2)^2

Also in this case, I should say that this is a linear regression model because for the definition, w0 , w1 and w2 are still linear in the expression. Even if there is a quadratic term for the independent variable x2.

Now , I have this question. A model like the following :

y = w1 x1 + w2 x2 + w3 x3 + w4 x1x2 + w5 (x2^3)

Is it still a linear model ? My first answer is yes , because for the definition the parameter terms are linear , but I'm not sure of it. Does anyone got any hint ?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
1
$\begingroup$

Theoretically, you do indeed have a linear model, yes.

You have a linear relationship between the dependent variable and your parameters. You would still be going through the process of performing linear regression, fitting a line through points via a linear combination of some regressors. However you have a non-linear equation due to the higher-order regressors you have manually inserted (x1x2 and x2^3).

This is enough to still call it a linear model generally speaking - see some useful answers in this thread.

In general I wouldn't say you have a linear model in the strictest sense at the end, as you are modelling linear relationship between your dependent variable y and non-linear combinations of your regressors: x, x2, and x3, but perhaps there could be an underlying feature to be observed which is exactly equal to x1x2, call it x4, and then you would have removed one of the non-linear covariates. Tyhe same goes for the final non-linear term.

Improve this answer
$\endgroup$
2
  • $\begingroup$ One question though ..in stats linear regression is defined as combination of independent variables...which is not in case of x^2, so is the terminology wrong in ML? $\endgroup$
    – DuttaA
    Jun 26, 2018 at 14:55
  • $\begingroup$ As I mentioned, there are linear combinations of weights/parameters. If y has a non-llinear relationship to $X_1$, let's say the true relationship is of order 4: $y \sim X^{4}_1$, then it would be a non-linear problem. But if we put $X^{4}_1 $ as a regression, we allow the model to express that non-linear relationship with a linear model. $\endgroup$
    – n1k31t4
    Jun 26, 2018 at 15:01
0
$\begingroup$

I do not think your terminology is correct. What you are talking of id Polynomial Regression which we generally use in Machine Learning.

Here are the definitions:

Linear Regression - Linear regression is a linear approach to modelling the relationship between a scalar response (or dependent variable) and one or more explanatory variables (or independent variables). The case of one explanatory variable is called simple linear regression. For more than one explanatory variable, the process is called multiple linear regression. This term is distinct from multivariate linear regression, where multiple correlated dependent variables are predicted, rather than a single scalar variable.

Polynomial Regression - Polynomial regression is a form of regression analysis in which the relationship between the independent variable x and the dependent variable y is modelled as an nth degree polynomial in x. Polynomial regression fits a nonlinear relationship between the value of x and the corresponding conditional mean of y.

The point to note is in Linear Regression it is a combination of independent variables. This definition is used in statistics, so I do not know whether we are using wrong terminology in ML.

Improve this answer
$\endgroup$
10
  • $\begingroup$ I read that the definition of linear regression is that we assume that the y (dependent variable) is a linear combination of parameters (w) and not (necessary) independent variables (x). So the last model I posted in the question should be linear if we follow this definition. But in this case also the polynomial regression should be linear , am I wrong ? If the right definition is the one you wrote , polynomial reg in ML fits a non linear model , is that right? $\endgroup$
    – Rubio95R
    Jun 26, 2018 at 15:08
  • $\begingroup$ @Rubio95R the definitions i gave are from Wikipedia and statistics so I am not sure if it applies to ML..(it should though) $\endgroup$
    – DuttaA
    Jun 26, 2018 at 15:18
  • $\begingroup$ @Rubio95R machinelearningmastery.com/… this site also says the same...I think in machine learning we assume the combinations to be independent $\endgroup$
    – DuttaA
    Jun 26, 2018 at 15:19
  • $\begingroup$ Here there is a ML definition I found : "Linear regression requires a linear model. No surprise, right? But what does that really mean? A model is linear when each term is either a constant or the product of a parameter and a predictor variable. A linear equation is constructed by adding the results for each term. This constrains the equation to just one basic form: Response = constant + parameter * predictor + ... + parameter * predictor Y = b o + b1X1 + b2X2 + ... + bkXk In statistics, a regression equation (or function) is linear when it is linear in the parameters. " $\endgroup$
    – Rubio95R
    Jun 26, 2018 at 15:26
  • $\begingroup$ @Rubio95R well see in statistics x1 and x2 are independent which is not in the case of x1 and x1^2...alsoif you have a term of (w1*x1)^2 you can write it as w1_new * x1^2...so whatever you do it remains linear only...kind of ambiguous $\endgroup$
    – DuttaA
    Jun 26, 2018 at 15:29
0
$\begingroup$

Yes , all the models you mentioned are linear models.Linear models are linear in parameters.

One important assumption in linear regression is that your parameters are not correlated (note: your parameters can be dependent though). When you have terms like x^2 and x1x2 , you may end up having highly correlated parameters which will create problems such as driving significant coeffs insignificant because of huge increase in the variance of the coefficients of the correlated parameters. This is just a consequence of having correlated parameters in your linear model.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.