I can use shap to extract important features for Dense NN. However, for CNN, I encountered two problems:

  1. the feature order may be messed up or combined after the filtering and pooling. Can I still use shap or other approach to show the important features for CNN?
  2. I tried to use shap, but the shap_summary_plot shews the bar plot to the left and the plot_size does not help to adjust it.
  • $\begingroup$ Could you provide a screenshot/code please? SHAP has nothing to do with the feature order actually ... $\endgroup$ Jan 8, 2021 at 20:01

1 Answer 1


There are these approaches you can consider to gain insights into feature importance for CNNs.

  1. CAM (Class Activation Map): CAM is a technique used to visualize which parts of an image are most important for a particular class prediction made by a CNNs. It highlights the regions of the input image that contribute the most to the final classification decision. This approach can provide valuable insights into which regions of the input image are significant for the CNNs's prediction.

  2. Layer-wise Relevance Propagation (LRP): LRP is an approach that assigns relevance scores to the input features based on the relevance of the output. It can help to attribute the contribution of individual input features to the final prediction, even after convolutions and pooling.

  3. Feature Visualization: You can try to visualize the filters learned by the CNNs in the early layers. These filters represent the features the CNNs is detecting. By visualizing the filters, you can get an idea of what the CNNs considers important.

Regarding the issue with the shap_summary_plot, you can try custom plotting. If the built-in shap_summary_plot is not giving you the desired results, you can also consider customizing the plot using matplotlib or plotly directly. This will give you more control over the plot layout and size.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.