I'm currently learning about neural networks, and it seems to me that there usually is no good theoretical explanation given for why certain architectures work; there is most of the times, no formal argument as to why certain architectures work. The explanation give by professors seems to be more that we devise architectures based on what we think might work; not on some formalized notion. As an example, we think GRUs work because they allow us to remember information from the past, and allows us to forget and remember as we chose. Well, who's to say simple skip level connections don't allow that? basically, is it the case that ANNs will remain blackboxes, and new techniques are basically architectures coming from gut feeling as to what might work?


You are right to guess that there is no solid theoretical way to design neural models. This is in part consistent with the free lunch theorem which says that there is no universal algorithm that can solve all problems efficiently.

The performance of neural models not only depends on the idea but also on the data. Batchnorm has a valid reason for normalising the input to each hidden layer in neural networks but it doesn't always improve the performance of models. One can find the limitations of current neural architecture for a given data and can come up with a fix that might improve the performance but it doesn't always mean that it will prove to be better.

As of now the field is more or less driven by an empirical approach rather than a theoretical one when it comes to designing new neural architecture.

| improve this answer | |
  • $\begingroup$ Thanks, this was really helpful! Kind of reinforced what I believed about the field. $\endgroup$ – user3676846 Mar 25 at 22:25
  • $\begingroup$ Good to hear that and welcome to stack exchange. If you found my answer helpful then can you 'upvote' and 'accept' my answer? $\endgroup$ – user1825567 Mar 26 at 3:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.