Wrong output multiple linear regression statsmodels

Question

I recently moved to python for data analysis and apparently I am stuck on the basics. I am trying to regress the parameters of the following expression: z=20+x+3*y+noise, and I get the right intercept but the x and y parameters are clearly wrong. What am I doing missing? Code below:

import numpy as np
import pandas as pd
import statsmodels.formula.api as smf 

# generate true values, and noise around them
np.random.seed(5)
x = np.arange(1, 101)
y = np.arange(1, 101)
z = 20 + x + 3* y + np.random.normal(0, 20, 100)

data = pd.DataFrame({'x':x, 'y':y, 'z': z})

lm = smf.ols(formula='z ~ x + y', data=data).fit()

# print the coefficients
lm.summary()

returns

where the x and y parameters are both 1.5, instead of being 1 and 3. What's wrong?

Answer 1

I think that you are seeing what you are seeing because the model sees the relationship of each set of points in your data frame, which are governed by the equation:

 z = 20 + x +3*y + noise 

But all the model sees is the resulting Z, NOT that equation which you know created z.

So it attempts to build a model which considers how Z was accomplished without knowing there was noise, while KNOWING that both x and y were in this equation because you explicitly told it the are in the model.

Based on this data. (at least this is what I got without a seed, when I ran your data..so it is close with probably different Z due to different noise)

x   y   z
1   1   32.824550
2   2   21.382597
3   3   80.615424
4   4   30.958157
5   5   42.192197
6   6   75.649622
7   7   29.815352
8   8   40.167267
9   9   59.752065
10  10  53.402601

Because x and y are the same for each point and because your formula has x + 3*y + noise , z is also equal to 4*x+ noise or 4*y+ noise or 2*x +2*y+ noise for each row. There are many ways to get the same change to Z with x & y in exact proportion plus some noise.

So the regressors are being assigned equal influence plus an equal share of the noise. It is the most parsimonious evaluation of x and y.

It is not what you expect knowing the equation, but it is the answer you should get. If you use a linear model which reduces variables of zero influent, you might even get a 0 or NA value for y.

To test that lm is working, simply reverse the order of y changing the relationship between x & y within that equations and you have a completely different result. I think the one you were looking to find.

y = np.arange(100,0,-1)


            coef    std err    t    P>|t|   [0.025  0.975]
Intercept   0.0439  0.000   119.009 0.000   0.043   0.045
x           1.2184  0.038   32.471  0.000   1.144   1.293
y           3.2130  0.038   85.629  0.000   3.139   3.288

You built your model right, but in this case the data set was not created in a way to test for what you hoped to see...but it was not wrong.