An abstract idea for the performance diffs between SLP and MLP

Question

Recently I am working on some predictive analytic which based on neural network.

When I tried some tests on MLP with one hidden layer or multiple hidden layers, the results showed that:

1. one hidden layer performance on prediction is always better than multiple hidden layers

2. the tests were executed both on Knime and R models, they gave me the same trends

Based on my knowledge, more hidden layers with more neurons should perform better?

Is there a principle on this about for which kind of dataset turns out this kind of result?

Or I may need some book / article / paper to read?

Do you have any abstract idea (not mathematical algorithms) for me, thanks!

UPDATE

I am working on a fraud detection dataset, which includes 1000 observations. and all the dimensions are numericed and normalized...

But I am actually asking for a general idea, for the general idea to choose algorithm for different dataset

Answer 1

More layers/more neurons does not necessarily mean you will get better performance. If your data is too simple or the number of observations is not that high, then adding more parameters (more layers/neurons) may result in overfitting the data. During training, the network will try to represent the training data as closely as possible. When there are a good number of units in the network, then the representation that is learned by the network will mainly model the general trend of the data faithfully. But, if there are too many neurons, then it is possible that some of the neurons will simply model noise in the training data which does not generalise to unseen data, resulting in worse performance.

Also if you are using a saturating nonlinearity as an activation function (i.e. sigmoids or tanh) then adding too many layers may result in vanishing gradients which will cause your network to train very slowly, or perhaps not at all.

Designing neural networks that work well is not very easy, and the best way to get an intuition for what will work is to experiment. One thing to try is to evaluate a range of sizes for your hidden layer(s). If you are using sigmoid or tanh as your activation function, I suggest you try using rectified linear units as well.