I was wondering how do we have to decide how many nodes in hidden layers, and how many hidden layers to put when we build a neural network architecture.
I understand the input and output layer depends on the training set that we have but how do we decide the hidden layer and the overall architecture in general?
Sadly there is no generic way to determine a priori the best number of neurons and number of layers for a neural network, given just a problem description. There isn't even much guidance to be had determining good values to try as a starting point.
The most common approach seems to be to start with a rough guess based on prior experience about networks used on similar problems. This could be your own experience, or second/third-hand experience you have picked up from a training course, blog or research paper. Then try some variations, and check the performance carefully before picking a best one.
The size and depth of neural networks interact with other hyper-paramaters too, so that changing one thing elsewhere can affect where the best values are. So it is not possible to isolate a "best" size and depth for a network then continue to tune other parameters in isolation. For instance, if you have a very deep network, it may work efficiently with the ReLU activation function, but not so well with sigmoid - if you found the best size/shape of network and then tried an experiment with varying activation functions you may come to the wrong conclusion about what works best.
You may sometimes read about "rules of thumb" that researchers use when starting a neural network design from scratch. These things might work for your problems or not, but they at least have the advantage of making a start on the problem. The variations I have seen are:
Create a network with hidden layers similar size order to the input, and all the same size, on the grounds that there is no particular reason to vary the size (unless you are creating an autoencoder perhaps).
Start simple and build up complexity to see what improves a simple network.
Try varying depths of network if you expect the output to be explained well by the input data, but with a complex relationship (as opposed to just inherently noisy).
Try adding some dropout, it's the closest thing neural networks have to magic fairy dust that makes everything better (caveat: adding dropout may improve generalisation, but may also increase required layer sizes and training times).
If you read these or anything like them in any text, then take them with a pinch of salt. However, at worst they help you get past the blank page effect, and write some kind of network, and get you to start the testing and refinement process.
As an aside, try not to get too lost in tuning a neural network when some other approach might be better and save you lots of time. Do consider and use other machine learning and data science approaches. Explore the data, maybe make some plots. Try some simple linear approaches first to get benchmarks to beat, linear regression, logistic regression or softmax regression depending on your problem. Consider using a different ML algorithm to NNs - decision tree based approaches such as XGBoost can be faster and more effective than deep learning on many problems.
