Can the feature importance scores of multiple ML algorithms be combined?

Question

Multiple Machine Learning algorithms are developed to rank some features. Is there an algorithm or statistical approach that can combine the ranking of each of these features into a final ranked list that takes the feature importance scores of the algorithms into consideration?

Answer 1

In recent years I have read different approaches on what you mention, with the argument of applying an ensamble on feature selection, the same way it is applied to model prediction (stacking models)

References: This one and This one

One of my favorites due to its simplicity is the following cited in the first link from Data Robot:

---

1 . Calculate the feature importance for the top N best models in the leaderboard against the selected metric. You can calculate feature importance by measures such as permutation or shap impact.

2. For each model with computed feature importance, get the ranking of the features.

3. Compute the median rank of each feature by aggregating the ranks of the features across all models.

4. Sort the aggregated list by the computed median rank.

5. Define the threshold number of features to select.

6 .Create a feature list based on the newly selected features.

---

So basically you are going to have a matrix of shape (n_features, n_models*), whose values are going to be the rank of importance of feature i in model m for i in {1,..,n_features} and m in {1,...,n_models}

*Note that you could also add as columns the importances given by SHAP or Permutation importance.

Then you have to calculate the median (or any other central tendency metric like mean) row-wise, i.e. the median of the rank of importances for each feature across all models, so finally you are able to establish a threshold for which you are going to filter the feature set.

This method has the advantage of being both an ensemble and model agnostic, since you are only using the rank of importances regardless of how that importance is being calculated for each method, but the disadvantage of defining an appropriate threshold.

Hope it helps!