Why do the deconv outputs of layer >= 2 of Zeiler&Fergus look so unrealistic?

Question

Reading the Zeiler&Fergus paper (my summary), I wonder how exactly they trained the deconv net. What was their data?

I think for one CNN which they want to analyze, they train exactly one deconv net (in contrast to training one deconv net per layer). The featuers (inputs) of the deconv net are the activations of the layer they want to analyze. The output they train them on are the activations that actually was the input of the layer they want to analyze. So although they have one deconv-net in total, they train it layer-wise. So for each training run, the weights of only one deconv layer are adjusted.

However, I wonder why the images look that unrealistic:

Is it gray because MSE is the training objective? Why aren't the first layer filter outputs gray then, too?

Answer 1

The reconstructed images from deconvnet represent the patterns in the original images which are very likely to strongly activate a particular featuremap at a particular convolution layer.

The gray portion in the reconstructed images shows that the values of those pixels in input images are not likely to strongly activate that featuremap.

There is no such gray portion in the first layer filters because first layer filter are representing the very simple patterns(e.g. edges) and those are very likely to present in input image at everywhere.