How to obtain the optimum model using Keras? Deep Learning

Question

There are too many parameters while building an artificial neural network. Some of which that comes to my mind are:

1. Number of layers
2. Types of layers
3. Number of nodes in each level of layer
4. Activation functions in each layer
5. Ordering of the layers
6. Different types of optimizers
7. Different types of loss functions
8. Batch size while fitting
9. Epochs while fitting
10. A lot of other things I never heard of (I suppose)

The lone fact that the number of layers might be in the order of tens and the number of nodes might be in the order of thousands implies that we can end up with billions of different combinations. Considering all the parameters in the list above, how can one know that their model is close to the optimum ?

The cases I study on does not require me to be too precise in terms of r2_score. In other words, it would not make a huge difference if the r2_score is 0.85 or 0.87. But the question is how do I know if I am real close to the saturation point of the r2_score for the given data, i.e. the optimum model would yield an r2_score of 0.90 and I am not stuck at 0.80?

Answer 1

A lot of this comes down to just trying different configurations, it can almost be seen as more as an art than a science. However, there are certain rules of thumb that can be applied based on existing research and models. Examples of this are that the Adam optimizer works well enough for most cases (point 6) and convolutional layers generally work better than others (e.g. dense layers) when working with images (point 2). It is therefore always good to see if you can find existing papers that work on problems that are similar to yours and see which approaches are used and what (doesn't) work(s) well.