1
$\begingroup$

My goal is to extract the feature importances from already trained random forest classifier and transfer them to another classifier. How this can be done? and How can access to modify feature_importances of Random Forest Classifier model? using scikit learn.

$\endgroup$
0
$\begingroup$

I'm not sure it's possible to "transfer" the feature importances from model to model in a Random Forest Classifier. Although I think there are two work-arounds you may be able to use.

The first option you have is to retrain the second model on the exact same training data. This will have the effect of both models having close to the same feature importances, though I think doesn't guarantee they will be exactly the same.

Another way would be to try knowledge distillation (https://arxiv.org/abs/1503.02531). Here you're basically making a target, or student, model train to not only match the hard labels, but also a set of soft labels, generated from a teacher network. In this case, the model that you want to transfer the knowledge to would be the student network. However, I believe this will involve writing a custom loss function, which I don't think is possible in sklearn. The paper I linked is in the context of neural networks so I also don't know if there would be any gaps in the performance between model types, as presumably, you would doing this with a random forest.

Just to confirm, I ran this short script that trains two models on the same data and it seems to be doing what you want it to:

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification


X, y = make_classification(n_samples=1000, n_features=4,
                           n_informative=2, n_redundant=0,
                            random_state=0, shuffle=False)

clf = RandomForestClassifier(max_depth=2, random_state=0)
clf.fit(X, y)
print('model 1 feature importances: ', clf.feature_importances_)

clf2 = RandomForestClassifier(max_depth=2, random_state=0)
clf2.fit(X, y)
print('model 2 feature importances: ', clf2.feature_importances_)

The results I got were:

model 1 feature importances:  [0.14205973 0.76664038 0.0282433  0.06305659]
model 2 feature importances:  [0.14205973 0.76664038 0.0282433  0.06305659]

Here you can see that the results were the exact same. I would imagine that the integrity of the transfer may change as the data changes. Make sure both models are initialized with the same random state if you want the feature importance values to match exactly.

$\endgroup$
1
  • 1
    $\begingroup$ Welcome to the site, Rohin! However, this is not the correct way to use warm_start. You need to call it repetitively on the same model object, e.g. on clf in your code. It does not transfer weights between different models. Regarding your example: You would get the same result if you set warm_start=False because clf2 is the same type of model as clf with identical hyperparameters. $\endgroup$ – Sammy Aug 12 '20 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.