Recently I am working on some predictive analytic which based on neural network.

When I tried some tests on MLP with one hidden layer or multiple hidden layers, the results showed that:

  1. one hidden layer performance on prediction is always better than multiple hidden layers

  2. the tests were executed both on Knime and R models, they gave me the same trends

Based on my knowledge, more hidden layers with more neurons should perform better?

Is there a principle on this about for which kind of dataset turns out this kind of result?

Or I may need some book / article / paper to read?

Do you have any abstract idea (not mathematical algorithms) for me, thanks!


I am working on a fraud detection dataset, which includes 1000 observations. and all the dimensions are numericed and normalized...

But I am actually asking for a general idea, for the general idea to choose algorithm for different dataset

  • $\begingroup$ What kind of data are you training the neural network on? $\endgroup$ Dec 20, 2016 at 2:37
  • $\begingroup$ @timleathart, please see the updated infos, thanks. $\endgroup$
    – cinqS
    Dec 20, 2016 at 2:46

1 Answer 1


More layers/more neurons does not necessarily mean you will get better performance. If your data is too simple or the number of observations is not that high, then adding more parameters (more layers/neurons) may result in overfitting the data. During training, the network will try to represent the training data as closely as possible. When there are a good number of units in the network, then the representation that is learned by the network will mainly model the general trend of the data faithfully. But, if there are too many neurons, then it is possible that some of the neurons will simply model noise in the training data which does not generalise to unseen data, resulting in worse performance.

Also if you are using a saturating nonlinearity as an activation function (i.e. sigmoids or tanh) then adding too many layers may result in vanishing gradients which will cause your network to train very slowly, or perhaps not at all.

Designing neural networks that work well is not very easy, and the best way to get an intuition for what will work is to experiment. One thing to try is to evaluate a range of sizes for your hidden layer(s). If you are using sigmoid or tanh as your activation function, I suggest you try using rectified linear units as well.

  • $\begingroup$ This is basically what I want, and the overfitting problem is what I am considering about. But do you have any infos to provide from a higher level of why it happens like this? thanks $\endgroup$
    – cinqS
    Dec 20, 2016 at 3:08
  • $\begingroup$ I have added an explanation how overfitting can occur in neural nets in my answer $\endgroup$ Dec 20, 2016 at 3:18
  • 1
    $\begingroup$ Thanks, and this really helped me. and the noise modelling because of too many unnecessary neurons will definitely affect the result. thanks $\endgroup$
    – cinqS
    Dec 20, 2016 at 3:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.