How are new neural network architectures 'discovered'

Question

This might not be on topic here, but fingers crossed.

There are all manner of neural network architectures out there, everything from convolutional networks to deep recurrent networks (and even deep recurrent convnets). Some network arcitecture development I can kind of see the intuition behind; recurrent networks allow for feedback loop type things to be taken into account. Other ideas seem to come out of left field and change the field. Yolo is a good example of this - I understand how it works, I've implimented it on my own dataset. But how did someone come to the set up that allowed yolo to work.

Is it just creativity, trying random architectures and new ideas to see what sticks? Is there some methodology that I could apply to take some steps towards a new ground-breaking architecture? (I understand that I won't actually make the next new architecture, but it's interesting to understand the process)

Answer 1

Think of neural networks as math, and ask yourself how is new math discovered? New math comes from a combination of theory, experimentation and feedback from community.

Instead of thinking about it terms of creativity vs randomness to see what new ideas stick; I prefer to think in terms of theory and experimentation.

The current methodology for new neural net discovery is no different than any other applied field of math/engineering.