1
$\begingroup$

I'm trying to optimize a neural network architecture for a particular problem, but there just seems to be so many hyperparameters that I'm concerned that there are much better options that I'm not exploring (e.g. I might be getting trapped in a local minima for hyperparams).

Essentially, I'd like some standard bounds for hyperparameter search. Ideally, if a person were to see the breadth of explored parameters, they might be reasonably certain to try another class of machine learning models. In particular, I am looking for concrete advice on at least

What size neurons to include in the search

  • The number of hidden layers, and the width of the layers
  • The optimizer and learning rates
  • Activation functions and loss functions

Thank you so much!

$\endgroup$
1
$\begingroup$

In designing an MLP architecture, we can restrict ourselves to 4-8 layers with 8-128 (power of 2) neurons per layer. In addition, we can assume recommended ReLU activations with He normal weight initialization and Adam or SGD with Nesterov momentum optimizers (see ipython notebook for comparison). Your loss function will depend on the problem: cross-entropy for classification and MSE for regression.

In order to avoid local optima in parameter search, it's better to use random search or bayesian optimization. For additional answers, see this post.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.