How to determine number of Neurons setup in Convolutional Neural Networks?

Question

Im new to deep learning and still learning on how to train my neural networks from the scratch. Sometimes I watch tutorial on YouTube or even online courses on the MOOC platforms. The base of a Convolutional Neural Networks usually has Conv2D and MaxPooling layers to make the input much more smaller and easy to be trained.

The thing is sometimes this tutorial online use the setup of Conv2D with higher number of neurons follows with other Conv2D with smaller one. Such as 1st layer is Conv2D(512...), 2nd layer is Conv2D(256....) and etcetera. Another tutorial setup is the increasing one, such as starts with Conv2D(16....), then Conv2D(32....) and increasing. These teacher doesn't tell us why they code the setup as so.

How do we know which setup to use? Is there any differences between them? Can't find on the net how to refer this case. If there's a paper that already described this case I would like to have it thanks.

Answer 1

A typical best practice is to read related papers or check github projects with information about similar architectures and how other ML engineers structured them. Usually, you can see some certain patterns in CNN architectures like increasing the depth along the pipeline (or number of channels) while decreasing width and height of results in each layer. Since the search space is huge and time so limited, it is usually better to rely upon already made architectures, and use, if possible, transfer learning to shorten training cycles. On the other hand, if you are doing pure research and want to create your top-notch, new state-of-the-art CNN architecture, you will need to carry out a very big effort on CNN architecture engineering as well as hyperparameter tuning, luckily on a systematic basis.

Answer 2

This is a design architecture choice and falls under the popular problem of hyperparameter choice in machine learning.

Typically the number of neurons increases as the network depth increases before decreasing again. The motivation behind this is to capture "local features" in the smaller layers before capturing more complex features of the data in the larger layers. Here's an example of the famous AlexNet that was one of the first neural networks to achieve human performance for image recognition:

In general "why" and "how" the number of neurons were chosen is really up to the designer and typically decided through empirical testing. The choice of them remains an open question in the field of machine learning.