At the time of writing the article, CNNs were not yet a particularly popular architecture for neural networks (and neural networks in themselves weren't as popular and common as today). In fact this article can be seen as the one who started the current age of deep learning as a default machine learning approach.
All this intro was for the soul purpose of saying that by "standard feed forward neural networks", the author meant neural networks that consist only out of dense layers (fully connected).
So to answer your last question first, in theory, anything a convolutional layer can do, a fully connected layer with the same number of input parameters and output parameters can also accomplish. i.e. you can produce a convolutional layer from a fully connected one (it will just have the same weights repeat themselves in a pattern of the same size as the convolution kernel), and you can also produce a lot more operations that a simple convolutional layer cannot perform. The simplest example can be a location dependent convolutional layer (where different areas of the image are convolved with different kernels). In reality, the search space for optimal parameters of such a layer would be so huge (and so non-convex), that it won't be able to converge.
The limitations on the convolutional layer are actually what helps her shine, which brings us to your first question. The "locality of pixel dependencies" is exactly what allows such a limited operation, to achieve such fine results. The meaning of the sentence is very simple, close pixels are very likely to be dependent on each other, so we can and should leverage this dependency to process them together (as is done with convolutional kernels).
Regarding the "stationary of statistics" phrase, I'm not 100% sure, but I think it refers to the fact that at small local patches, there is a higher probability of finding recurring patterns (the smaller the patch size, the smaller the possible variance of the patch). However as I said, I'm not sure about this one.