Activation Maps using Tensorflow Object Detection API

Question

I am trying to create visual explanations such as GradCAM from a trained object detection model. In order to implement the algorithm I need to access intermediate tensors and calculate gradients to the feature maps. The frozen inference graph seems to omit a lot of information such as tensor names, after using the exporter script (exporter_main_v2.py), therefore I am not able to access a specific tensor.

Using the "training model" (before converting to pb), I am also struggling to access specific tensors, because the model is not plain Keras/Tensorflow, but instead parts of the model are abstracted away in classes such as SSDMetaArch, SSDMobileNetFeatureExtractor etc. which hide the actual implementation of the model.

Any idea how to approach this problem?

Answer 1

In the end it turned out to be easier than originally thought. Using the detection model of type SSDMetaArch, we can recreate the model as a keras model, by specifying an input layer and using the individual building blocks of the detection model, i.e. feature extractor and box predictor.

input_layer = tf.keras.layers.Input(input_shape)
feature_maps = detection_model.feature_extractor(input_layer)
outputs = detection_model._box_predictor(feature_maps)
outputs['feature_maps'] = feature_maps  # add feature_maps to outputs to access them with GradientTape
cam_model = tf.keras.Model(inputs=[input_layer], outputs=outputs)

Now the gradients from any output neuron to the feature maps can be calculated using GradientTape. In the following example the gradient from detection score of the bounding box with the highest confidence to the feature maps is calculated.

with tf.GradientTape() as tape:
    input_image = image[None, ...]
    predictions = cam_model(image)
    feature_maps = predictions['feature_maps']
    tape.watch(feature_maps)
    idx = tf.argmax(predictions['class_predictions_with_background'][0])[query_class]
    output_neuron = predictions['class_predictions_with_background'][0, idx, 0]
grads = tape.gradient(output_neuron, feature_maps)