I am referring to plain neural networks, MLPs. I got to read the paper by Glorot and Bengio (2010), Understanding the difficulty of training deep feedforward neural networks.
Therein I read an interesting statement: ``Deep learning methods aim at learning feature hierarchies with features from higher levels of the hierarchy formed by the composition of lower level features''.
What does that mean in the framework of MLPs? As far as I could understand, such a features extraction and abstraction process is made feasible by the use of convolutional layers, hence more advancer Deep Learning models.