4
$\begingroup$

in my project, I am using asuite of shallow and deep learning models in order to see which has the best performance on my data. However, in the pool of shallow machine learning models, I want to be able to compare the coefficients of each regression model between each other. Example: I have

  • Lasso
  • Ridge
  • ElasticNet
  • AdaBoostRegressor
  • GradientBoostRegressor

I am aware that I could get the coefficients of Lasso, Ridge, and ElasticNet from model.coef_ and model.intercept_ from sklearn. However, AdaBoostRegressor does not have this, but rather, it has weights assigned to each of the estimators. Similarly, GradientBoostingRegressor of sklearn does not even have weights at all. How am I to compare the coefficients between each of the regressive models I am using??

$\endgroup$
  • $\begingroup$ Why do you want to do this? By definition different models work in different ways: assigning coefficients is not the only way and even for those which do the coefficients might not be comparable. $\endgroup$ – Erwan Jan 26 at 14:13
  • $\begingroup$ @Erwan I am interested to see the weights each model will assign each feature I have in my data, and see areas where the models assign different weights to some feature Xi for example, while they might all agree on assigning low weight for example to some feature Xj $\endgroup$ – Perl Jan 26 at 15:56
3
$\begingroup$

Weights across different types of models are not always comparable, so I think that it would make more sense to do this kind of comparison not across different types of model but within a single type of model varying:

  • the hyper-parameters (if any),
  • the set of instances (e.g. selecting different subsets randomly),
  • the set of features.

I'd recommend in particular varying the instances: if the weight of a feature tends to change a lot depending on the training set it's a sign of overfitting.

If the goal is to compare the importance of some specific features for different models, it's probably more reliable to directly evaluate how the model performs with/without the feature(s).

| improve this answer | |
$\endgroup$
3
$\begingroup$

It's not actually possible to directly compare model coefficients. What you might do that would make more sense is to compare similar metrics.

A good start would be to learn about explainability metrics that are comparable across models : LIME, SHAP... etc. (see here : https://christophm.github.io/interpretable-ml-book/) to see how models reacts on different features.

| improve this answer | |
$\endgroup$
  • $\begingroup$ The book looks very interesting, thanks for the link! $\endgroup$ – Erwan Jan 26 at 23:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.