I have built a neural network for approximating a certain function and decided on a metric how to evaluate the performance. Now, where do I start with optimizing to get the optimal result? I need to decide on number of layers, hidden neurons, learning rate, learning algorithms and all other hyperparameters. What is the usual procedure and order in which to optimize all network's parameters? I guess I am looking for some kind of general guidelines to follow to obtain the best possible performance
My initial suggestion is to look in the literature for networks which have been implemented to do solve similar problems and start from there. Then you should just try a bunch of different values for the parameters and see what works in your case.
See also this Cross Validated question: https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw