1
$\begingroup$

I'm optimizing the parameters for a single layer MLP. I've chosen to vary 4 parameters: hidden layer size, tolerance, activation, and regularization weights. Each of these has 4 possible values it can take (4^4 = 256 combinations).

So the question is, how does one determine that a set of parameters are statistically significantly better than another?

My stats is a little rusty, but my first thought was n-way ANOVA with 4 factors and 4 degrees of freedom in each factor. Is there something better?

Improve this question
$\endgroup$
2
  • 5
    $\begingroup$ Welcome to the site! People usually just choose the one that performs the best on the validation set, assuming that the sample is large enough to achieve statistical significance. However, I can't imagine testing 256 hyperparameter combinations on a realistic data set so yours might be small, in which case statistical significance may indeed be a consideration... which in turn calls into question the choice of using a neural network. Just out of curiosity, what is the "tolerance"? $\endgroup$
    – Emre
    Commented Nov 28, 2017 at 4:11
  • $\begingroup$ Sigma for improvement of the solution, see "tol" here: scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier. The data set size is ~13k, I'm sampling 5k at a time to learn from. It seems like most of them perform similarly, but their learning times can vary quite a bit, (9s-100s). It seems like I'd still want to perform a test to determine which is the best in this regard. Does 4 way ANOVA make sense? $\endgroup$
    – Braaedy
    Commented Nov 28, 2017 at 4:22

1 Answer 1

Reset to default
1
$\begingroup$

I too was in your place when I started using Neural networks. There are so many hyper-parameters to choose, and each take on many values. Like Emre said, you need to check the model which is giving best metric score on your data (Cross validation set). The parameter values of that model will be your optimized values. You can also check this link

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.