I am trying to perform linear regression and I want to analyse the available features beforehand. The task is to predict the value of a house. Some of them might have a high impact on the label, others are irrelevant... This is my code which performs the analysis and actual regression:

X_train, X_test, Y_train, Y_test = train_test_split( X, Y, test_size=0.2, random_state=1)
featureScores = SelectKBest(f_regression, k=10).fit(X_train, Y_train).scores_
model = LinearRegression()
reg = model.fit(X_train, Y_train)
df = pd.DataFrame({'actual':abs(model.coef_ / np.max(model.coef_)), 'feature analysis':featureScores/np.max(featureScores)})

Here are 2 different charts as they change quite a bit if I change the random_state (Why? This should not happen that much I think):

Image1 Image2

Feature 0 is the index of the row, so it has no impact on the label. Feature analysis aswell as regression model ignore it, perfect. But then there is e.g. feature 12: It contains the living space of the house, which is a very high factor for the price. Feature analysis got this right, however the regression model does not really use this information.

What could I be doing wrong?


1 Answer 1


You don't appear to have taken the scale of the data, and thus the coefficients, into account. If e.g. living space is measured in square feet, then the coefficient will be rather smaller than if living space is measured in square yards. See this stats.SE question for computing the standardized coefficients, and plot those (again normalized to sum to 1 if you like).

You may still see fairly different results. One culprit may be multicollinearity: while f_regression is a univariate test, the LinearRegression will balance correlated features against each other. In the extreme case that a feature is duplicated in your dataset, both of those columns will have the same f_regression score (I believe), but the coefficients for them in the LinearRegression can be any two values that sum to the "true" effect of the column.

  • $\begingroup$ Thnaks! The living space is measured in m^2, so it is usually between 50 and 200. Feature 10 is the year when the house was built and feature 11 the year when it was sold. So both contain values like 2010, 1995, ... The coefficients of those 2 are also far apart despite being on the same scale. This is what it looks like on normalized data, way better :) imgur.com/a/7dNcFE7 $\endgroup$
    – hecad57571
    Mar 9, 2022 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.