In the overwhelming number of works devoted to the neural networks, the authors suggest arhitechure in which each layer is a numbers of neurons is power of 2

what are the theoretical reasons(prerequisite) for this choice?


Deep Neural Networks are usually trained on GPUs to speed up training time. Using power of two for the network topology follows the same logic as using power of two for image textures in computer games.

The GPU can take advantage of optimizations related to efficiencies in working with powers of two. (see https://gamedev.stackexchange.com/questions/26187/why-are-textures-always-square-powers-of-two-what-if-they-arent)

  • $\begingroup$ Thank you, maybe you know the article devoted to the study of the effects of these tradition in terms of machine learning theory? $\endgroup$ – Roosh Jan 20 '17 at 17:36

It is just an arbitrary choice. You have to choose one number and the order of magnitude matters, but not the exact value. Powers of two just feel natural.

If you don't think so: Evaluate it on a given architecture. Lower the number of neurons from a power of two to a smaller number. If the time increases, you've proven me wrong.

  • $\begingroup$ You propose empirical method for disproof your hypothesis. How about mathematical proof consistency of the hypothesis? What if 256 neurons are too small(underfit), and 512 too many(overfit)? Is it optimal to use the power of two if it leads to negative consequences? $\endgroup$ – Roosh Sep 11 '17 at 6:21
  • $\begingroup$ @Roosh The only possible benefit of powers of two is speed due to cache alignment or something similar. Writing a proof about real hardware is close to impossible, especially when you don't know all parts. $\endgroup$ – Martin Thoma Sep 11 '17 at 6:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.