scikit-learn: feature analysis differs heavily from model coefficients

Question

I am trying to perform linear regression and I want to analyse the available features beforehand. The task is to predict the value of a house. Some of them might have a high impact on the label, others are irrelevant... This is my code which performs the analysis and actual regression:

X_train, X_test, Y_train, Y_test = train_test_split( X, Y, test_size=0.2, random_state=1)
featureScores = SelectKBest(f_regression, k=10).fit(X_train, Y_train).scores_
model = LinearRegression()
reg = model.fit(X_train, Y_train)
df = pd.DataFrame({'actual':abs(model.coef_ / np.max(model.coef_)), 'feature analysis':featureScores/np.max(featureScores)})
df.plot(kind='bar')
plt.show()

Here are 2 different charts as they change quite a bit if I change the random_state (Why? This should not happen that much I think):

Feature 0 is the index of the row, so it has no impact on the label. Feature analysis aswell as regression model ignore it, perfect. But then there is e.g. feature 12: It contains the living space of the house, which is a very high factor for the price. Feature analysis got this right, however the regression model does not really use this information.

What could I be doing wrong?

Answer 1

You don't appear to have taken the scale of the data, and thus the coefficients, into account. If e.g. living space is measured in square feet, then the coefficient will be rather smaller than if living space is measured in square yards. See this stats.SE question for computing the standardized coefficients, and plot those (again normalized to sum to 1 if you like).

You may still see fairly different results. One culprit may be multicollinearity: while f_regression is a univariate test, the LinearRegression will balance correlated features against each other. In the extreme case that a feature is duplicated in your dataset, both of those columns will have the same f_regression score (I believe), but the coefficients for them in the LinearRegression can be any two values that sum to the "true" effect of the column.