2
$\begingroup$

I use a brute-force mechanism to determine optimal hidden layers/neurons by incrementing the layers/neurons by 1 up to some maximums and then picking the optimal counts from the best performing model. My question here is about the starting point for the hidden neuron count of this "brute-force" process. What are some good mechanisms for determining what neuron count to start at such that I am not likely to miss some minimum optimal count?

For example:

Say I have 8 inputs that are NOT linearly related. If I take (number of inputs: 8) + 1 (as some post suggest) for my starting point of incrementing the number of neurons within the layers, would I likely be missing the optimal neurons count as it may be less than 9 neurons?

The brute-force mechanism I am using sets a max number of neurons per hidden layer, so if I have 8 inputs, and I started with a range of 9 to 30 neurons per hidden layer, I would have 9 neurons in the first hidden layer which would increment to 30 and then roll back to 9 when starting to test with a second hidden layer:

Cycle 1 Hidden Layers = 1 Hidden Neurons = 9 ... Hidden Layers = 1 Hidden Neurons = 30

Cycle 2 Hidden Layers = 2 Hidden Neurons = 9 ... Hidden Layers = 2 Hidden Neurons = 30

Cycle 3 Hidden Layers = 3 Hidden Neurons = 9 ... Hidden Layers = 3 Hidden Neurons = 30

I admit this has inefficiencies as each layer may not need the same number of neurons, but the point of this post is to try and improve this mechanism by minimizing the number of layers/neurons combinations that need to be tested to determine the "optimal" counts.

EDIT:

So just to clarify the question is about determining what a good minimum starting point would be if using the brute-force mechanism outlined above regardless of data domain (hence the input + 1 example).

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Hi there,this question has been asked many times, kindly have a quick search .. $\endgroup$
    – Aditya
    Commented Sep 23, 2018 at 13:58
  • 1
    $\begingroup$ stats.stackexchange.com/questions/181/… $\endgroup$
    – Aditya
    Commented Sep 23, 2018 at 14:00
  • $\begingroup$ Noted that these questions are for determining a starting point of layers/nodes to try, but not determining what a good minimum starting point would be if using the brute-force mechanism outlined above. $\endgroup$
    – user7226068
    Commented Sep 23, 2018 at 14:20

1 Answer 1

Reset to default
1
$\begingroup$

There is no hard-and-fast rule for this.

The number of hidden nodes you should have is based on a complex relationship between

  • Number of input and output nodes

  • Amount of training data available

  • Complexity of the function that is trying to be learned

  • The training algorithm

  • Too few nodes will lead to high error for your system as the predictive factors might be too complex for a small number of nodes to capture

  • Too many nodes will overfit to your training data and not generalize well

To conclude,

  • There is no hard and fast rule. You just have to keep trying with different number of layers to see which one works best...

  • In neural networks, model architecture is an art that you can master with some experience and domain knowledge.

Check this answer too...

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ So your answer is simply start with 1 neuron per hidden layer and increment up as there is no good rule for picking a starting node count that won't miss lower node counts that would be more optimal. $\endgroup$
    – user7226068
    Commented Sep 23, 2018 at 14:19
  • $\begingroup$ I too started the same way and then jumped by 10's,50's.. Currently I have them set as first 25% of the input and increment by 5% till it hits 50-55% $\endgroup$
    – Aditya
    Commented Sep 24, 2018 at 5:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.