I'm reading the ImageNet Classification with Deep Convolutional Neural Networks paper by Krizhevsky et al, and came across these lines in the Intro paragraph:

Their (convolutional neural networks') capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse.

What's meant by "stationarity of statistics" and "locality of pixel dependencies"? Also, what's the basis of saying that CNN's theoretically best performance is only slightly worse than that of feedforward NN?


At the time of writing the article, CNNs were not yet a particularly popular architecture for neural networks (and neural networks in themselves weren't as popular and common as today). In fact this article can be seen as the one who started the current age of deep learning as a default machine learning approach.

All this intro was for the soul purpose of saying that by "standard feed forward neural networks", the author meant neural networks that consist only out of dense layers (fully connected).

So to answer your last question first, in theory, anything a convolutional layer can do, a fully connected layer with the same number of input parameters and output parameters can also accomplish. i.e. you can produce a convolutional layer from a fully connected one (it will just have the same weights repeat themselves in a pattern of the same size as the convolution kernel), and you can also produce a lot more operations that a simple convolutional layer cannot perform. The simplest example can be a location dependent convolutional layer (where different areas of the image are convolved with different kernels). In reality, the search space for optimal parameters of such a layer would be so huge (and so non-convex), that it won't be able to converge.

The limitations on the convolutional layer are actually what helps her shine, which brings us to your first question. The "locality of pixel dependencies" is exactly what allows such a limited operation, to achieve such fine results. The meaning of the sentence is very simple, close pixels are very likely to be dependent on each other, so we can and should leverage this dependency to process them together (as is done with convolutional kernels).

Regarding the "stationary of statistics" phrase, I'm not 100% sure, but I think it refers to the fact that at small local patches, there is a higher probability of finding recurring patterns (the smaller the patch size, the smaller the possible variance of the patch). However as I said, I'm not sure about this one.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.