0
$\begingroup$

I have basic understanding of how to perform linear regression with sklearn and statsmodels. There are several questions that I would like to ask regarding Linear Regression (OLS estimator) :

  • Is standardization always necessary for OLS estimator? I am still learning a bit of theoretical aspect of regression analysis. Standardization is mentioned in my data science class but almost None throughout my study in regression analysis. If no, when do I apply standardization (also do I need to standardize the predicted variable)?

  • I tried comparing both, in terms of evaluation metrics(mse) the results are relatively similar, but in terms of coefficient it is very different (due to different scales of predictors). If this is the case how to compare the significance of a predictor?

  • I tried comparing the result with statsmodel OLS on unstandardized data. Coefficient result is different by huge margin and on top of that mean squared error is worse with statsmodel than with sklearn model. However, sklearn model is less informative in terms of statistical information, for example if I want to remove a variable based on p-value from regression result. Is this supposed to happen(different result)?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

1) No. You can standardize data, but you don‘t need to. Often you don‘t want this because you like to see the effect of some $x$ on $y$ in „real units“ (not standard deviations).

2) In case you standardize $X$, the coefficients must of course change as the X‘s are now no longer real units but standard deviations. The advantage here is that you can compare coefficients in terms of the magnitude or importance (larger absolute value, more important). This is sometimes called „beta coefficients“ or „standardized coefficients".

The model fit may be a little better with standardization, but the main restriction in OLS is the linear model structure. You can relax this assumption by using "Generalized Additive Models" GAM or introducing polynomials.

3) I don‘t understand your question here and there is no code, but it is likely that you did something wrong. OLS like $y=\beta X + u$ simply requires solving

$$ \hat{\beta}=(X'X)^{-1}X'y.$$

Never drop variables due to high p-value (it summarizes confidence bands you could say). A variable may still be useful and important for the model even with high p-value. If you want to do feature/variable selection, use "shrinkage" via Lasso.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I just realized my mistakes. I forgot to add the bias term (hence inconsistent result) in my OLS model on statsmodel (the default behaviour of statsmodel OLS assumes no intercept term). Thanks for the answer though. $\endgroup$
    – Yohanes Alfredo
    Oct 31, 2019 at 11:00
  • $\begingroup$ "You can standardize data, but you don‘t need to. Often you don‘t want this because you like to see the effect of some x on y in „real units“ (not standard deviations). " It is a controvesial statement $\endgroup$
    – Subhash C. Davar
    Mar 11, 2021 at 15:32
  • $\begingroup$ In causal models (e.g. in econometrics) data usually is not standardized. The literature is full of such models. So the statement is not controversial at all. Can you clarify what you find controversial here? $\endgroup$
    – Peter
    Mar 11, 2021 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.