Skip to main content
1 of 3
SQLServerSteve
  • 371
  • 1
  • 4
  • 11

The point of the layer depth and gradual pyramidal reduction is to build up a hierarchy of invariant representations, each more complex than those of the prior levels. For example, at the lowest level, a convolutional may be able to pick out noteworthy arrangements of pixels; at the next level, it can condense these into particular spots, basic shapes, edges, etc.; then at higher levels it can recognize increasingly large and complex objects. I'll borrow an example from Gerod M. Bonhoff's thesis1 on Hawkins' Hierarchical Temporal Memory (HTM), which is a closely related concept, one that also makes use of receptive regions to build invariant representations. At higher levels, the filtering process allows a convolutional or HTM to assemble individual lines and shapes into objects like "dog's tail" or "dog's head"; at the next stage they can be recognized as a "dog" or perhaps a particular variant, like "German shepherd."

This is made possible not only by the stacking of multiple layers, but the divisions of neurons within them into separate receptive regions. The receptive regions mimic actual neuronal "cell assemblies" and cortical columns that learn to fire together in groups; this enables clustering around particular types of objects, while the additional layers allow them to be related together into objects of increasing sophistication. The decrease in spatial dimensions in the example you cited reflects the narrowing of the receptive regions as we move up the pyramid; the third dimension (i.e. depth within a layer, as opposed to the depth of the layers) increases in tandem so that we can provide a wider choice of invariant representations to select from at each stage. If we simply narrowed the pyramid at each stage along every dimension, eventually we'd be left with just a narrow range of objects to choose from; taken far enough, it might just leave us with a single node at the top reflecting a single yes-no choice between "is this a dog or not?" This more flexible design allows us to choose more combinations of the previous layer's invariant representations.

This type of design is inspired by various biologically plausible structures found in actual organisms, such as the eyes of cats. If what I've said here isn't clear enough to answer your question, I can add a lot of further detail, including more examples, some based on actual organs of that kind.

1See pp. 26-27, 36 76 Bonhoff, Gerod M., Using Hierarchical Temporal Memory for Detecting Anomalous Network Activity. Thesis delivered March, 2008 to the faculty of the Air Force Institute of Technology at Wright-Patterson Air Force Base, Ohio.

SQLServerSteve
  • 371
  • 1
  • 4
  • 11