5
$\begingroup$

Many works use 2-hidden-layer neural networks for classifying MNIST handwritten digits sets.

To improve accuracy, other techniques (dropout, ReLU.. etc) have been used without increasing the number of hidden layers.

Is there any reason not to use more than three hidden layers? for example, overfitting?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Empirically, the network performance does not increase much for a fully-connected network on MNIST when you add layers, but you can probably find ways to improve it on networks with 3+ hidden layers, such as data augmentation (e.g. variations of all inputs translated +-0..2 pixels in x and y, roughly 25 times the original data size, as a start).

I don't think this idea is pursued very far in practice, because CNNs offer a much better performance increase for the effort required. You hit the point of diminishing returns earlier with a basic MLP (around 96-97% accuracy) than you can reach easily with a CNN (around 99% accuracy).

The theory basis for this difference is not obvious to me, but very likely yes this is related to over-fitting. The weight sharing and feature pooling in a CNN is very effective way of processing image data for classification tasks, and avoids over-fitting by reducing the number of parameters, whilst re-using the parameters for the task in a way that makes very good sense given the nature of the inputs.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ I would like to add that MNIST is a very 'easy' task, the data is very clean and easy to generalize on, it lies on a very low dimensional manifold which reduces the need for very deep layers. That said, I think blowing up the number of fully connected, dense layers and adding very strong regularization could get you to a better performance, close to CNNs. $\endgroup$
    – Jan van der Vegt
    Commented Aug 11, 2017 at 12:53
  • $\begingroup$ @JanvanderVegt: Just heaping on regularisation to a deep fully-connected network will not generalise as well as CNN at image tasks, you would need a lot of data augmentation too. CNN generalisation is partially a regularisation effect (through shared weights), but the structural match to the problem is a big part of it too. $\endgroup$
    – Neil Slater
    Commented Aug 11, 2017 at 17:00
  • $\begingroup$ Thank you so much for your comments. However, I wonder why the accuracy of 2-hidden-layer is better than that of 3-hidden-layer neural network. It is what I have experienced. Even if I didnt add any regularization techniques, I dont know the reasons of decreasing accuracy. I think it is not overfitting problem because training error and test error are similar. $\endgroup$
    – S.Lim
    Commented Aug 11, 2017 at 23:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.