Below is a gradcam implementation for a standard image classifier :

from keras.applications.vgg16 import VGG16, preprocess_input, decode_predictions
from keras.preprocessing import image
import keras.backend as K
import numpy as np
import cv2
import sys 
model = VGG16(weights="imagenet")
img_path = sys.argv[1]
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

preds = model.predict(x)
class_idx = np.argmax(preds[0])
class_output = model.output[:, class_idx]
last_conv_layer = model.get_layer("block5_conv3")

grads = K.gradients(class_output, last_conv_layer.output)[0]
pooled_grads = K.mean(grads, axis=(0, 1, 2))
iterate = K.function([model.input], [pooled_grads, last_conv_layer.output[0]])
pooled_grads_value, conv_layer_output_value = iterate([x])
for i in range(512):
    conv_layer_output_value[:, :, i] *= pooled_grads_value[i]

heatmap = np.mean(conv_layer_output_value, axis=-1)
heatmap = np.maximum(heatmap, 0)
heatmap /= np.max(heatmap)

img = cv2.imread(img_path)
heatmap = cv2.resize(heatmap, (img.shape[1], img.shape[0]))
heatmap = np.uint8(255 * heatmap)
heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)
superimposed_img = cv2.addWeighted(img, 0.6, heatmap, 0.4, 0)
cv2.imshow("Original", img)
cv2.imshow("GradCam", superimposed_img)

I would like to implement a similar gradmap on my MobileNetSSD architecture (model is stored as caffemodel)

net = cv2.dnn.readNetFromCaffe('MobileNetSSD_deploy.prototxt.txt', 'MobileNetSSD_deploy.caffemodel')
image = cv2.imread(PATH_FOLDER + 'images/img_01.jpg')
(h, w) = image.shape[:2]
blob = cv2.dnn.blobFromImage(cv2.resize(image, (300, 300)), 0.007843, (300, 300), 127.5)

The mobilenetSSD network may detect many classes in one image, so ideally I would like to build one heatmap per class.

Here is how I get the class_ids present in the image :

output= net.forward()

for i in np.arange(0, output.shape[2]):
    confidence = output[0, 0, i, 2]
    if confidence > 0.8:
        class_idx = int(output[0, 0, i, 1])

In order to build the gradmap, I am trying to retrieve class_idx (done),class_output and last_conv_layer

To get last_conv_layer_output, I did :

last_conv_layer_output = net.forward(outputName = 'conv17_2_mbox_conf')
class_output = net.forward(outputName = 'mbox_conf_softmax')[:,7]

grads = K.gradients(class_output , last_conv_layer_output)[0]

Problem here is that my gradient is None, meaning that either class_output or last_conv_layer_output (or both) are miscalculated. I am a bit lost on how to debug this?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.