aggregation of feature importance

Question

I have more of a conceptual question I was hoping to get some feedback on. I am trying to run a boosted regression ML model to identify a subset of important predictors for some clinical condition. The dataset includes over 100000 rows, and close to 1000 predictors. Now, the etiology of the disease we are trying to predict is largely unknown. Thus, we likely don’t have data on many important predictors for the condition. That is to say, as a prediction model, any model I come up with is going to do a rather poor job predicting the outcome. However, the primary aim here is not about prediction, but rather to identify important variables which we can then target more directly in future analyses. So I am trying to use the ML model as a variable selection tool.

Normally we can get a sense of the model performance by evaluating its metrics on a new dataset – for example by using nested cross validation or a train-test split. But rather than evaluating the model’s metrics, my primary interest here is to evaluate the consistency by which the different predictors are being chosen (i.e. the consistency of the feature importance list). So essentially, I think what I want to do is to randomly split the database (say use 60% of the data), run CV to tune the hyperparameters, and then using the best hyperparameters train the model on the full 60% and get the feature importance. Then I would repeat the same process X number of times, each time using a different randomly chosen 60% sample. This would give me X number of tables of feature importance, one from each run. But is there a way to then somehow “merge” all these feature importance tables to get a sense of how stable the selection process is across the different runs? Or are there better ways altogether to do this? Thanks a lot!

Answer 1

This is an interesting one and there is not a "one-fits-all" answer to it. If I break down your question into two major parts, I would say:

Choosing important variables

• Domain expert: It is always helpful to have an idea from the domain expert on what variables matter the most, especially in your case that you have 1000 variables to choose from. Given the size of your dataset, it would really reduce the time of processing.
• Intelligent variable importance measures: there are so many ways you can limit your variables to the ones that really matter, starting from simple correlation and covariance metrics to a Bayesian one. There are packages out there that will do this for you. For example, if you are programming in R, Boruta is one of the ones I use a lot (read here). Having said that, you can always use the ones from sklearn. So I am sure you appreciate that this is all pre-modelling before you go through any ML modelling.

Once you have an understanding of how variables matter and to what extent, I would limit the ML to the top 100 maybe? This will save you heaps of time and effort.

ML modelling
I like your approach on the train-test split and in fact, it is what I do most of the time. You can define a dataframe to hold the importance measures from your ML model each time you run the model and append the new ones to it. this way, at the end of say 20 rounds of modelling, you will have 20 measures for each variable which you can then average and rank.

The whole process should give you an idea of what variables really matter and then I would get into "hyperparameter optimization" after I have figured what variables to use.

Answer 2

1.) If your aim is to find the most relevant features the first thing you should do is feature engineering using Domain Knowledge. It is the most effective feature selection technique out there and in done properly, does not require any extra feature selection techniques.

2.) After you do the above step, if you want to get a measure of "importance" of the features w.r.t the target, mutual_info_regression can be used. It will give the importance values of all your features in on single step!. Also it can measure "any kind of relationship" with the target (not just a linear relationship like some techniques do).

3.) If you further want to do feature selection, you can try one the wrapper based techniques like RFECV, forward/backwards selection techniques etc. Personally I would go with RFECV!

I would say focus most your effort on step 1 as it will definitely give you best results.