What's the intuition behind nonlinear predictors in multivariate regression?

Question

I'm learning statistical learning with the well known ISLR (Introduction to Statistical Learning with Applications in R) and doing the exercises, right now in the linear chapter regression. Despite being linear regression, non linear transformations are also mentioned because of the high bias issues.

It's quite straightforward to understand the meaning of a linear regression model with multiple predictors, e.g.:

Y = β0 + β1X1 + β2X2 + β3X3

So as X1 increases by one unit, predicted response Y will increase by β1 units (assuming X2 remains constant), same for the rest of predictors.

Also for regression with interaction terms:

Y = β0 + β1X1 + β2X2 + β3X1 X2

Which in this case indicates there's a synergy between X1 and X2 predictors, removing the additive assumption.

I'm also getting familiar with F-statistic, t-value or p-value for assessing the statistical significance of the predictors and decide whether or not there's an actual relationship with the predicted variable.

In the applied exercises section for this chapter, one of the exercises used with a dataset with several values, contains the following question:

Try a few different transformations of the variables, such as log(X), sqrt(X), X^2. Comment on your findings.

How should one try nonlinear transformation of variables following an intuition? It's easy to get a sense of this for simple regression with a single predictor (just by plotting the dataset), but what's the technique here without just randomly try stuff, considering that there are several predictors to try them all?

Moreover, given the following hypothetical model:

Y = β0 + β1X1 + β2X2 + β3X3 + β4X1^2 + β5log(X2)

That is, a predictor having both a linear and non linear relationship with the response? What can be a real-world example to understand this? As said, this would be very easy with a single predictor, but I don't find an intuition with multivariate regression.

Answer 1

Not quite sure if this is what your question actually is. However, when you have a data generating process such as $y = 3 + 2x$ (with some zero-mean "gaussian noise"), your model could look like:

$$ y = \beta_0 + \beta_1 x + u,$$

and you will find the $\beta_0 = 3$ and $\beta_1 = 2$.

When you have a data generating process such as $y=3+2x+3x^2$ you need to do a "basis expansion" in your model to capture the additional complexity of the function and you could write a model such as:

$$ y = \beta_0 + \beta_1 x + \beta_2 x^2+ u.$$

The second model will capture the non-linear (quadratic) DGP "perfectly".

df = data.frame(x1=seq(-10,20,by=0.2))

df$v = df$x1^2 
df$err = rnorm(nrow(df), mean = 0, sd = 1)
df$y = df$v + df$err 

plot(df$x1,df$y)

reg1 = lm(y~x1,data=df)
summary(reg1)
lines(df$x1,predict(reg1,newdata=df),col="red")

reg2 = lm(y~poly(x1,2,raw=T),data=df)
summary(reg2)
lines(df$x1,predict(reg2,newdata=df),col="blue")

Answer 2

I will give you an intuitive example to try out yourself - that will showcase the significance of interaction variables.

Let us say you had to predict a person's height by age and Gender:

X = Age, Gender y = height

You have two modelling options:

Option 1 - y_hat = b0 + (b1 x age) + (b2 x gender)

So, you have age and height as independent predictors for someone's height.

Nothing earth shattering yet - this is what you would generally expect to do.

Option 2 - y_hat = b0 + (b1 x age) + (b2 x gender) + (b3 x age x gender)

This is a much more expressive model (notice the interaction term between age and gender - with coefficient b3)

If gender is male (0) - this breaks down to:

y_hat = b0 + (b1 x age) + (b2 x 0) + (b3 x age x 0), or further

**y_hat = b0 + (b1 x age) ** - I

If gender is female (1):

y_hat = b0 + (b1 x age) + (b2 x 1) + (b3 x age x 1), or further

y_hat = (b0+b2) + (b1+b3) x age, or further

y_hat = B0 + (B1 x age) - II

With the interaction variables - you can see that the regression from Option 1 is actually broken down automatically into two separate regression models in Option 2. The two models are represented by I and II under Option 2. It is likely that the Model from Option 2 is much more expressive than Model 1