How to get feature importance from a keras deep learning model?

Question

In case of scikit-learn's models, we can get feature importance using the relevant attributes of the model.
I've been working on a RNN, using LSTMs for text embedding. Is there any way to get feature importance of various features from the finalized model?

Answer 1

Although this thread is pretty old to sustain any active thoughts, I still humbly feel that @user1877600 response is not in sync with the query presented by @soham_dhole.

If that's the case, leaving my comment for future reference:

Neural networks are more like a black box algorithm when it comes to gaining insights about the features as in other machine learning algorithms however there are passive ways to get the needful few of which are discussed below.

• Gradients Importance - idea is to check the contribution of each single input feature on the final prediction output. The contribution in our case is given by the value of the gradients obtained from the differentiation operation of the input sequences on the forecasts.

With Tensorflow, the implementation of this method is only 3 steps:

1. use the GradientTape object to capture the gradients on the input
2. get the gradients with tape.gradient: this operation produces gradients of the same shape of the single input sequence (time dimension x features)
3. obtain the impact of each sequence feature as average over the time dimension. For more info, please take a look at: Feature Importance with Time Series and Recurrent Neural Network

• Permutation Feature Importance - We do this with a for-loop of size N where N is the number of features we have. For each feature we wish to evaluate, we infer our validation metric (let's say MAE) with that feature column randomly shuffled. If this feature column is important to our LSTM model, then the MAE (our validation metric) will become worse for that for-loop step. After our for-loop, we display bars equal to the size of how much MAE worsened without each feature, which is the importance of each feature.
• LOFO (Leave One Feature Out) - Importance calculates the importance of a set of features based on a metric of choice, for a model of choice, by iteratively removing each feature from the set, and evaluating the performance of the model, with a validation scheme of choice, based on the chosen metric.

Thanks!

Answer 2

There is plenty of methods to calculate feature importance. I recommend trying two of them LIME and SHAP.

I don't want to copy-paste material and tutorial provided by the author so please refer to these two repositories.