CNNs: understanding feature visualization Channel Objectives (SOLVED)

Question

I'm trying to follow a paper on deep NN feature visualization using beautiful examples from the GoogLeNet/Inception CNN. see: https://distill.pub/2017/feature-visualization/

The authors use backpropagation to optimize an input image to maximizes the activation of a particular (Inception) neuron/feature, or entire channel.

For example, Inception Layer 4a, Unit 11 is feature 12 of 192 from the 1x1 convolution path of Inception Layer 4a before filter concatenation (see: https://distill.pub/2017/feature-visualization/appendix/googlenet/4a.html#4a-11).

For Layer 4a 1x1 convolution the shapes are:

# Layer 4a
input:    [14,14,480]
output:   [14,14,512]

# 1x1 convolution
kernel:   [1,1,480]   # total 192 kernels
output:   [14,14,192] # channels [0..192] of Layer4a output
Layer4a slice:    tf.slice( layer4a_output, (0,0,0), (14,14,192) ) 

# Layer4a Unit 11
layer4a_unit_11 = tf.slice(layer4a_output, (11,0,0), (1,1,1)) # numpy [11,1,1]

In a related article, the authors state (see: https://distill.pub/2018/building-blocks/) ,

"We can think of each layer’s learned representation as a three-dimensional cube. Each cell in the cube is an activation, or the amount a neuron fires. The x- and y-axes correspond to positions in the image, and the z-axis is the channel (or detector) being run."

Furthermore, they offer a diagram which super-imposes the cube of Layer4a over the input image with the (x,y) axis overlaying the image itself.

I understand that the Neuron Objective is the input image that produces the highest activation for Layer 4a, Unit 11 which can be found at index=[11,0,0] of Layer 4a output=[14,14,512]. In this case, (x,y)=[11,0]. Each [1,1,480] kernel generates a feature map of shape=[14,14,1] with a total of 196 activations.

kernel => channel or feature map and activation => neuron or feature.

Question

But what is the intuitive concept of the (Positive) Channel Objective? In this example, Unit 11 sits in the same channel as 14x14=196 other neurons, but the channel objectives for all these neurons are different. If the optimized image for the Channel Objective maximizes the sum of neuron activations for channel 0, (e.g. slice=[14,14,0] of 192 1x1 convolutions or 512 total layer 4a channels) wouldn't it be the same for all 192 neurons in the same channel? Obviously, by the examples we see this is not true.

How does the Channel Objective relate to the Neuron Objective for Unit 11?

ANSWER

I understand that the Neuron Objective is the input image that produces the highest activation for Layer 4a, Unit 11 which can be found at index=[11,0,0] of Layer 4a output=[14,14,512].

This is where my understanding went off the rails. Layer 4a Unit 11 is actually channel/feature 12 of 192 for the 1x1 convolution. It is NOT the 12 of 196 neuron of channel 1. My fault for confusing 192 channels with 196 neurons/channel.

Instead, as mentioned the in answer, Unit 11 is a single neuron in channel 11, usually located near the center, e.g. Neuron Objective is (x,y,z)=(7,7,11) and Channel Objective is (x,y,z)=(:,:,11)

Answer 1

As you noticed, the activations on mixed4a for a normal sized input is [14,14,512]. Intuitively, these dimensions correspond to [x_position, y_position, channel].

When we talk about "channel" and "neuron" objectives for visualization of channel 11, we mean:

• Neuron objective: maximize activations[7, 7, 11]
  • maximize channel 11 in one position (generally middle)
• Channel objective: maximize tf.reduce_sum(activations[:, :, 11])
  • maximize channel 11 everywhere

This diagram from feature visualization also goes over the difference: