I'm optimizing the parameters for a single layer MLP. I've chosen to vary 4 parameters: hidden layer size, tolerance, activation, and regularization weights. Each of these has 4 possible values it can take (4^4 = 256 combinations).

So the question is, how does one determine that a set of parameters are statistically significantly better than another?

My stats is a little rusty, but my first thought was n-way ANOVA with 4 factors and 4 degrees of freedom in each factor. Is there something better?

  • 5
    $\begingroup$ Welcome to the site! People usually just choose the one that performs the best on the validation set, assuming that the sample is large enough to achieve statistical significance. However, I can't imagine testing 256 hyperparameter combinations on a realistic data set so yours might be small, in which case statistical significance may indeed be a consideration... which in turn calls into question the choice of using a neural network. Just out of curiosity, what is the "tolerance"? $\endgroup$
    – Emre
    Nov 28 '17 at 4:11
  • $\begingroup$ Sigma for improvement of the solution, see "tol" here: scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier. The data set size is ~13k, I'm sampling 5k at a time to learn from. It seems like most of them perform similarly, but their learning times can vary quite a bit, (9s-100s). It seems like I'd still want to perform a test to determine which is the best in this regard. Does 4 way ANOVA make sense? $\endgroup$
    – Braaedy
    Nov 28 '17 at 4:22

I too was in your place when I started using Neural networks. There are so many hyper-parameters to choose, and each take on many values. Like Emre said, you need to check the model which is giving best metric score on your data (Cross validation set). The parameter values of that model will be your optimized values. You can also check this link


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.