On looking at various machine learning methods at the scikit-learn site http://scikit-learn.org/stable/modules/classes.html , it appears that some modules such as linear regression ( http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html ) provide coefficients (coef_), others such as AdaBoostRegressor ( http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostRegressor.html ) provide feature_importances_ while some e.g. BaggingRegressor ( http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingRegressor.html ) do not provide either of these.

Why this difference? Are coefficients similar to feature_importances_ to assess the contribution of a variable in prediction? How to assess the feature importance for modules where neither of these is available e.g. in BaggingRegressor (link above) and BernoulliNB ( http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.BernoulliNB.html )?


The main difference between Linear Regression and Tree-based methods is that Linear Regression is parametric: it can be writen with a mathematical closed expression depending on some parameters. Therefore, the coefficients are the parameters of the model, and should not be taken as any kind of importances unless the data is normalized.

On the other hand, non-parametric methods have very different ways of measure the importance. In layman terms, measuring the importance of variables on a tree can be done by checking how close they appear to the root node. In ensembling methods, like bagging, one can compute the importance of a variable as the average among the ensemble, like in this stackoverflow answer.

The main difference is, then, the fact that parametric models have, through their parameters, a way of showing the importance of the variables, while non parametric models need some extra work.

  • $\begingroup$ Did I answer your question? $\endgroup$ – David Masip May 3 '18 at 8:43
  • $\begingroup$ Apologies for missing to answer your comment earlier. Yes, your answer clarifies a lot. Thanks. $\endgroup$ – rnso Oct 4 '19 at 5:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.