0
$\begingroup$

I was solving one quiz question on Coursera and I found an interesting question.

If you double the value of a given feature (i.e. a specific column of the feature matrix), what happens to the least-squares estimated coefficients for every other feature? (assume you have no other feature that depends on the doubled feature i.e. no interaction terms).

My questions are -

  1. I think, the other coefficients will stay constant. If so, can someone tell me the logical explanation behind it?
  2. What about the coefficient of the scaled feature ? will it be equal to m/2 assuming we doubled the feature and existing weight was m ?
  3. what if interaction terms are included as well ? what will happen there?

So, collectively I can say that Regression coefficients are independent of change of origin but not of scale. will it be correct ? If we shift, the coefficients remain unchanged. However the coefficients will change when we scale?

$\endgroup$
1
  • $\begingroup$ Consider the matrix form of the OLS estimator. $\endgroup$
    – Dave
    Jan 1 at 17:48

1 Answer 1

1
$\begingroup$
  1. Other coefficients will stay constant because scaling a feature doesn't change the correlation between the feature and the dependent variable. The intuition might be that "rests" of prediction between y~x and y~2x are the same (ax = (a/2)(2x)) and other coefficients don't have to change.
  2. Yes, it will be equal to m/2 because as I mentioned (ax = (a/2)(2x)) to do the same job (still best possible prediction).
  3. It depends on the type of interaction, but if the interaction is multiplying features, the coefficient should be divided by 2 (ax1x2 = (a/2)(2x1)(x2)).

Let's check it out in an experiment:

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

np.random.seed(1)
x1, x2 = np.random.rand(100), np.random.rand(100)
df = pd.DataFrame({'x1': x1, 'x2': x2, 'x12':x1*x2, 'y': 2*x1 + 3*x2 + np.random.normal(0, 0.1, 100)})

model = LinearRegression().fit(df[['x1', 'x2', 'x12']], df['y'])

print('Initial')
print("Coefficients:", np.round(model.coef_,3))
print("Intercept:", round(model.intercept_,3))

df['x1'] *= 2
df['x12'] = df.x1*df.x2
model = LinearRegression().fit(df[['x1', 'x2', 'x12']], df['y'])

print()
print('x1 doubled')
print("Coefficients:", np.round(model.coef_,3))
print("Intercept:", round(model.intercept_,3))

We get:

Initial
Coefficients: [ 2.03   3.032 -0.095]
Intercept: 0.002

x1 doubled
Coefficients: [ 1.015  3.032 -0.048]
Intercept: 0.002

And that is consistent with out expectations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.