Neural Network - Adjust number of hidden layers and neurons

Question

I am using KNIME MLP Neural Network learner (If you are not familiarized with KNIME think of that like a package which implements Neural Network to a set of data).

The thing is you can tune the number of hidden layers, the number of iterations and neurons per layers, so here comes the first question: How to know a priori values for these parameters? For instance, for Random Forest 500 trees is supposed to be a standard to test its performance in the first stages.

Second, I know in general add more layers and/or neurons does not ensure any enhance in the prediction, sometimes even the opposite happen. Someone could give some theoric insights respect this topic; why by adding more layers the prediction might get deteriorated?

Thanks in advance!

Sergi

Answer 1

Sorry for not knowing exact number for the first question, but my point is to start from the most simple model, rather than any random parameter. Then use a grid search for the parameter by validation.

As for the second question, the error of a model comes from two parts, variance and bias. Bias comes from lack of complexity of your model. For example, there will always be a large bias when using linear model to fit the distance v.s. the shooting angle of a canon. This can be overcome by using a more complex model, adding layers to the NN for example.

And variances comes from being too sensitive to the data, even to the noise in the data. When there is a high variance due to using a too complex model, it is called over-fitting. You may take a look at this page(over-fitting on wiki). And using more layers/neurons is using a more complex model, which means the model may suffer from high variance. This can be overcome by lower the model complexity(which may increase the bias) or adding training data.

Basically if you got constant number of data, then you have to find somewhere where both bias and variance is not too high.