1
$\begingroup$

I'm using a deep CNN with the ReLU activation function. When visualizing the layers (each with 32 filters), several of the filtered images are zeros.

I am trying to reason why this may be happening? What would cause the entire image to be zero? I can understand having a sparse image (given the ReLU), but what would cause all zeros?

I hope this is clear and has sufficient detail to answer appropriately.

$\endgroup$
  • $\begingroup$ Are the filtered images zero across all input images or just specific ones? $\endgroup$ – matthiaw91 Jun 7 at 21:59
  • $\begingroup$ Looking at the output for the first convolutional layer, for example, 7 out of 32 images are zero images. Just not sure if this is normal? My CNN does what I want in the end; just wondering why the image is entirely zeros (i.e. after activation every pixel is zero, or perhaps it dies during backpropagation)? I have yet to look at the filters themselves. $\endgroup$ – Shinobii Jun 8 at 2:59
  • $\begingroup$ That depends. What I actually wanted to know if these e.g. 7 out of 32 feature maps are zero for every input image form the training set or just a few? $\endgroup$ – matthiaw91 Jun 8 at 14:37
  • $\begingroup$ I get what your saying; all I have to visualize are the global outputs that are fed to the next layer (i.e. sum of the convolved data after applying ReLU). Your basically asking me to check the convolved data prior to the summing? I guess, one really negative filter would kill the entire image (regardless of the other images). I will take a closer look at my data. $\endgroup$ – Shinobii Jun 9 at 2:30
2
$\begingroup$

In principle, it is to be expected that several feature maps result in zero output all over the image. What the convolutional layers are doing, essentially, is to move a stencil across the image and give a high value where this stencil shape is found in the image, and zero everywhere else. Each of the feature maps, i.e. each of the channels in the convolutional layers, corresponds to a stencil, the lower layers encode simple shapes, the higher layers more complex ones.

If a channel outputs zero all over, it simply means that this feature was not detected in the image.

As an example, consider an MNIST classifier. One feature map may have learned to look for straight horizontal lines. When the model is presented with a hand-written "7", the straight horizontal lines will be detected in the top part of the image and maybe also in the middle. So the feature map will give high values in the top part and maybe in the middle as well, while it will be zero everywhere else.

If the model is presented with a hand-written "0", there is usually no horizontal lines and therefore the feature map will be zero everywhere.

So if a feature map is zero everywhere, it usually means that the particular feature is not found in the current input, but it might be present in other images.

If, however, a feature map is all zeros for all input images, i.e. the channel in the convolutional layer is zero everywhere for all 60k MNIST examples, then there might something wrong with the network and you might want to apply batch normalization or dropout.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Right, this makes sense. Especially without any dropout whatsoever, some filters will die while other thrive (to sum it up). Thank you for clarifying this! $\endgroup$ – Shinobii Jun 9 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.