What are the default values of nodes and internal layers in Neural Network models?

Question

What is the default number of internal layers and internal nodes in training a neural network?

My data has 62 observations with roughly 200 predictors. I have a target variable with two classes and implemented a neural network with one internal layer and one internal node without repeats. Also, I tried with two internal layers, with 5 internal nodes in one, and 2 internal nodes in second layer. I want to find the accuracy, first, on default values and then I will try to optimize the model performance.

What is the criterion to choose the number of layers and internal nodes in a neural network training model? In the case of Random Forest, we can choose to try to be roughly equal to the square root of the number of predictors.

Answer 1

One potential approach can be iterative design of a neural network architecture such as Multi-Layer Perceptron (MLP) as described in the following post:

https://stats.stackexchange.com/questions/238637/deep-neural-network-tuning-hyperparameters

We can restrict ourselves to 4-8 layers with 8-128 (power of 2) neurons per layer. In addition, we can assume recommended ReLU activations with He normal weight initialization and Adam or SGD with Nesterov momentum optimizers.

In order to avoid overfitting on a small dataset, it is important to add l1 or l2 regularization (weight decay) and a dropout layer (e.g. with keep probability of 0.5).

We can then use cross validation with random search or bayesian optimization to choose the best architecture as described in the cross validated article above.

Answer 2

There are websites that explain these pretty well.

Deciding on the number of neurons in the hidden layer(s)

From https://www.r-bloggers.com/selecting-the-number-of-neurons-in-the-hidden-layer-of-a-neural-network/:

The most common rule of thumb is to choose a number of hidden neurons between 1 and the number of input variables.

Deciding on the number of layers of hidden layers

From https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw:

For most problems, one could probably get decent performance (even without a second optimization step) by setting the hidden layer configuration using just two rules: (i) number of hidden layers equals one; and (ii) the number of neurons in that layer is the mean of the neurons in the input and output layers.

Hope that answers your question!