0
$\begingroup$

I am trying to find out what is optimum number of neurons that can be used in MNIST dataset(60,000 training and 10,000 testing data). I build a single hidden layer model using keras,with relu activation but when I increase the neurons in hidden layer the accuracy improves slightly nearly .4% per 100 neurons ,even if I use 2000 neurons instead of 784 neurons it doesn't has any considerable change.I thought that increasing neurons above certain limit will decrease the classification accuracy because of overfitting.but It doesn't seem to look happening I even tried using 2000 neurons but the accuracy doesn't decrease on the test data.exceeding this limit I think is not possible for me because it can crash my computer.

Is hit and trial the only way to find the optimum number of neurons in a MLP hidden layer. How can I further tweak the parameters , can it be some other hyper parameters that I can play with like intialzing hidden layer weights initially.or adding more layers.(but adding more layer doesn't help much) I tried it.

Improve this question
$\endgroup$
1
  • $\begingroup$ If the input is having 784 neurons then the hidden layer(s) shouldn't have more than twice your number as this is a simple task, What you should rather do is like deepening your network by increasing the hidden layer count.. rather than increasing the neurons number exponentially, Na there is no rule to decide that as it's a Hyperparameter and it comes with experience or trials.. $\endgroup$
    – Aditya
    Mar 5, 2018 at 1:55

1 Answer 1

Reset to default
2
$\begingroup$

If you train long enough and have "too many" hidden layer units, then you will eventually have over-training. Usually, the goal is to find the smallest number of hidden units that can do reasonably well, as that will generally give you good generalization. In fact, there is the "bottle neck theory" that attempts to show this effect (https://openreview.net/forum?id=ry_WPG-A-)

For more on comparisons between flat hidden layer networks versus convolutional networks, see:

http://conx.readthedocs.io/en/latest/MNIST.html#A-multi-layer-perceptron-network-for-MNIST-classification

Improve this answer
$\endgroup$
3
  • $\begingroup$ How will I know the best hyperparameters? $\endgroup$
    – Boris
    Mar 5, 2018 at 12:59
  • $\begingroup$ It depends on how you define "best." Do you want to learn all of the training patterns, or do very well on your test set? Once you decide, you can search the space of hyperparameters. Conx also has a way of running a set of experiments over combinations of hyperparameter values: conx.readthedocs.io/en/latest/Experiments.html $\endgroup$
    – Doug Blank
    Mar 5, 2018 at 13:11
  • $\begingroup$ I don't want to use conx $\endgroup$
    – Boris
    Mar 6, 2018 at 5:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.