How to extract the reason behind a prediction using TensorFlow?

Question

I created a CNN using TensorFlow2 and trained it as a binary classifier. Is there a way to extract the influence of each pixel upon the prediction?

I am trying to obtain a mask similar to the following:

Answer 1

[updated]

You are looking for GradCam.

Code:

1. https://fairyonice.github.io/Grad-CAM-with-keras-vis.html

2. https://tf-explain.readthedocs.io/en/latest/index.html

Paper: https://arxiv.org/abs/1610.02391

This is a good resource I found https://christophm.github.io/interpretable-ml-book/

Answer 2

The tf-explain package supports many interpretability methods. In particular, Grad CAM can "visualize how parts of the image affects neural network's output by looking into the activation maps":

from tf_explain.callbacks.grad_cam import GradCAMCallback

model = [...]

callbacks = [
    GradCAMCallback(
        validation_data=(x_val, y_val),
        class_index=0,
        output_dir=output_dir,
    )
]

model.fit(x_train, y_train, batch_size=2, epochs=2, callbacks=callbacks)