1
$\begingroup$

Below is a gradcam implementation for a standard image classifier :

from keras.applications.vgg16 import VGG16, preprocess_input, decode_predictions
from keras.preprocessing import image
import keras.backend as K
import numpy as np
import cv2
import sys 
model = VGG16(weights="imagenet")
img_path = sys.argv[1]
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

preds = model.predict(x)
class_idx = np.argmax(preds[0])
class_output = model.output[:, class_idx]
last_conv_layer = model.get_layer("block5_conv3")

grads = K.gradients(class_output, last_conv_layer.output)[0]
pooled_grads = K.mean(grads, axis=(0, 1, 2))
iterate = K.function([model.input], [pooled_grads, last_conv_layer.output[0]])
pooled_grads_value, conv_layer_output_value = iterate([x])
for i in range(512):
    conv_layer_output_value[:, :, i] *= pooled_grads_value[i]

heatmap = np.mean(conv_layer_output_value, axis=-1)
heatmap = np.maximum(heatmap, 0)
heatmap /= np.max(heatmap)

img = cv2.imread(img_path)
heatmap = cv2.resize(heatmap, (img.shape[1], img.shape[0]))
heatmap = np.uint8(255 * heatmap)
heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)
superimposed_img = cv2.addWeighted(img, 0.6, heatmap, 0.4, 0)
cv2.imshow("Original", img)
cv2.imshow("GradCam", superimposed_img)
cv2.waitKey(0)

I would like to implement a similar gradmap on my MobileNetSSD architecture (model is stored as caffemodel)

net = cv2.dnn.readNetFromCaffe('MobileNetSSD_deploy.prototxt.txt', 'MobileNetSSD_deploy.caffemodel')
image = cv2.imread(PATH_FOLDER + 'images/img_01.jpg')
(h, w) = image.shape[:2]
blob = cv2.dnn.blobFromImage(cv2.resize(image, (300, 300)), 0.007843, (300, 300), 127.5)
net.setInput(blob)

The mobilenetSSD network may detect many classes in one image, so ideally I would like to build one heatmap per class.

Here is how I get the class_ids present in the image :

output= net.forward()

for i in np.arange(0, output.shape[2]):
    confidence = output[0, 0, i, 2]
    if confidence > 0.8:
        class_idx = int(output[0, 0, i, 1])
        ...

In order to build the gradmap, I am trying to retrieve class_idx (done),class_output and last_conv_layer

To get last_conv_layer_output, I did :

last_conv_layer_output = net.forward(outputName = 'conv17_2_mbox_conf')
class_output = net.forward(outputName = 'mbox_conf_softmax')[:,7]

grads = K.gradients(class_output , last_conv_layer_output)[0]

Problem here is that my gradient is None, meaning that either class_output or last_conv_layer_output (or both) are miscalculated. I am a bit lost on how to debug this?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.