I have a dataset with 10 features.
I've computed the feature importance using permutation importance with cross-validation from eli5, after fitting an extremely randomized trees (ET) classifier form Scikit learn.
I fitted 10 different ETs using only one feature at a time and computed the mean cross-validation score using the same CV scheme.
I noticed while the order of importance of the features that I get from the permutation importance matches the order of features when I rank them using the mean CV scores I get when fitting model with only one feature at a time, there is one feature that permutation importance classes at the very bottom while its mean CV scores when the model is fitted using it alone is the second one.
Suppose that from 1. the order of the features is feature2, feature4, feature5, feature1, feature9, feature8, feature7, feature10, feature3, feature6. When using one feature at a time and computing the mean cv score, I noticed that the mean cv score of feature 3 is the second-best, although the importance feature is suggesting it is a weak feature.
I would appreciate if anyone can help me understand how to interpret this and which approach seems to be more trustworthy.
Thank you.