0
$\begingroup$

I am attempting to determine the most useful bands of a multiband image classification (i.e. Red, Green, Blue, Near Infrared, etc. used for classifying pixels) and wrote the following function to build a decision tree. It uses sci-kit learn's Decision Tree Classifier with entropy as the split criterion. Finally, it uses the feature_importances_ function to calculate the importance of each band:

def make_tree(X_train, y_train):
    """prints a decision tree and an array of
    the helpfulness of each band"""
    dtc = DecisionTreeClassifier(criterion='entropy')
    dtc.fit(X_train, y_train)
    
    tree.plot_tree(dtc)
    plt.show()
    
    importances = dtc.feature_importances_
    large_to_small_idx = np.argsort(importances)[::-1]
    for idx in large_to_small_idx:
        print(f"Band {idx + 1}: {importances[idx]}\n") 

I assumed that since the splitting criterion on the decision tree was set to entropy that feature_importances_ would also be calculated as some form of entropy information gain. However, in sci-kit learn's documentation it mentions how the feature importance is actually calculated:

The importance of a feature is computed as the (normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance.

Is this an issue or is the feature importance essentially still being calculated based on entropy? If this is not a good way to calculate feature importance based on entropy, is there a way to tweak feature_importances_ or some other method I am missing to do this? Thanks for the help!

$\endgroup$

1 Answer 1

0
$\begingroup$

The entropy criterion is used by the CART algorithm to build the DT itself, by evaluating which split is actually the best (greedily) to split on.

So, it's not directly related to feature importance which, in case of DTs, is computed as the reduction in impurity brought by a feature. This is not an error, it's just by design: actually, it is an extra capability that DTs have.

You can also estimate feature importance with Random Forests and Extra Trees which should provide more accurate results since they compute that from an ensemble of models. Indeed, the way it's computed is still based on impurity reduction, which you can think of being a quantification of how much a feature improves the model's performance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.