What can we understand from max-activation generated images?

Question

There are several approaches to generate psychedelic images, providing maximum activations for individual neirons in convolutional neural networks. For example there is a lot of them there https://app.slack.com/client/T040HKJE3/threads/thread/C04655480-1581762147.250800?cdn_fallback=2 or a bit in https://arxiv.org/abs/1311.2901 (M. Zeiler) or in https://www.youtube.com/watch?v=ghEmQSxT6tw But what can we get from them?

E.g. here does presence of 6 fish’es pieces means there are mainly 6 different positions of fishes which the net saw on 1000 train images? Or this is rather caused by some corner effects or patter size? Like 224 source pixels/100 = 2.2 fishes horizontally.

I also don't understand well how they are obtained, so can't conjecture what aspects they can depict.

There is approach when we find parts of train/validation images which gives strongest activations. This is perfectly understandable for me.

Answer 1

Those are the activation maps of the learned features.

In the case of this specific model, the filters learn that the "fishy" parts of the image:

• Head-ish and dorsal fin-ish sections that define convex hulls

• Scale-ish textures insides the convex hulls and surrounded by water-ish textures outside

• Orange-ish and gray-ish colors insides the convex hulls and blue-ish colors outside

• The head-ish section tend not to be in the corners of an image (since people frame pictures and people think faces are interesting)

Answer 2

What I can add after some more reading and thinking, is that "image, providing maximum activations for a neiron" implies that the net (by means of backpropagation) will try to go through most efficient way and even through several ways in parallel if possible.

1. Convolution of previous layer's neirons recognizing eye, body and fin will be maximum when they are at right positions relatively to each other. So the net is limited with minimum size of a pattern.

2. Convolution coefficients are something like -0.5, 1, 0.5 (I simplify 3*3 convolution to 1D). This means like "not here", "ideally here", "ok if here". So if previous layer's neiron has > 0 at both places 1 and 2, so the image will contain maybe 2 fins, this will give even higher activations. So generally the net will try to push maximum patterns into a given space (e.g. 224*224 pixels).

3. Maybe on infinite input image we would see identical repeating patterns, but in real life paddings at corners and so on limit them and make different.

So now I think positions on the generated images don't mean that they were such on source images, maybe a bit, indirectly, as a side effect of adjusting to corner effects. But the colors and pattern sizes has to reflect source images.

It's interesting why they are so noisy... First layers' convolutions are usually quite smooth...