I'm currently learning about neural networks, and it seems to me that there usually is no good theoretical explanation given for why certain architectures work; there is most of the times, no formal argument as to why certain architectures work. The explanation give by professors seems to be more that we devise architectures based on what we think might work; not on some formalized notion. As an example, we think GRUs work because they allow us to remember information from the past, and allows us to forget and remember as we chose. Well, who's to say simple skip level connections don't allow that? basically, is it the case that ANNs will remain blackboxes, and new techniques are basically architectures coming from gut feeling as to what might work?
1 Answer
You are right to guess that there is no solid theoretical way to design neural models. This is in part consistent with the free lunch theorem which says that there is no universal algorithm that can solve all problems efficiently.
The performance of neural models not only depends on the idea but also on the data. Batchnorm has a valid reason for normalising the input to each hidden layer in neural networks but it doesn't always improve the performance of models. One can find the limitations of current neural architecture for a given data and can come up with a fix that might improve the performance but it doesn't always mean that it will prove to be better.
As of now the field is more or less driven by an empirical approach rather than a theoretical one when it comes to designing new neural architecture.
-
$\begingroup$ Thanks, this was really helpful! Kind of reinforced what I believed about the field. $\endgroup$ Mar 25, 2020 at 22:25
-
$\begingroup$ Good to hear that and welcome to stack exchange. If you found my answer helpful then can you 'upvote' and 'accept' my answer? $\endgroup$ Mar 26, 2020 at 3:32