I'm trying to optimize a neural network architecture for a particular problem, but there just seems to be so many hyperparameters that I'm concerned that there are much better options that I'm not exploring (e.g. I might be getting trapped in a local minima for hyperparams).
Essentially, I'd like some standard bounds for hyperparameter search. Ideally, if a person were to see the breadth of explored parameters, they might be reasonably certain to try another class of machine learning models. In particular, I am looking for concrete advice on at least
What size neurons to include in the search
- The number of hidden layers, and the width of the layers
- The optimizer and learning rates
- Activation functions and loss functions
Thank you so much!