3
$\begingroup$

I have my ANN trained on MNIST dataset. Hidden layer has 128 neurons and input layer has 784 neurons. This gave me an accuracy of 94%. However when I added one more layer with 64 neurons in each then the accuracy significantly reduced to 35%. What could be the reason behind this.

Edit : Activation function : sigmoid. 521 epochs.

Improve this question
$\endgroup$
3
  • $\begingroup$ What is the activation function you are using? $\endgroup$
    – DuttaA
    Oct 28 '18 at 11:26
  • $\begingroup$ @DuttaA sigmoid $\endgroup$
    – Pink
    Oct 28 '18 at 12:33
  • $\begingroup$ Could you provide both training and test accuracy in both cases $\endgroup$
    – seanv507
    Oct 29 '18 at 15:37
2
$\begingroup$

The reason is that by adding more layers, you've added more trainable parameter to your model. You have to train it more. You should consider that MNIST data set is a very easy-to-learn dataset. You can have to layers with much less number of neurons in each layer. Try $10$ neurons for each to facilitate the learning process. You can reach to $100%$ accuracy.

Improve this answer
$\endgroup$
12
  • $\begingroup$ It's also a very small dataset. $\endgroup$
    – Matthieu Brucher
    Oct 28 '18 at 12:53
  • 1
    $\begingroup$ Yes! $50$ thousand is very smal for deep-learning purposes. $\endgroup$
    – Green Falcon
    Oct 28 '18 at 13:00
  • $\begingroup$ I added 10 neurons in both hidden layers. Trained for 100 epichs and the accuracy is 21% $\endgroup$
    – Pink
    Oct 28 '18 at 17:17
  • $\begingroup$ Epoch or iteration? $\endgroup$
    – Green Falcon
    Oct 28 '18 at 17:40
  • $\begingroup$ Epochs.......... $\endgroup$
    – Pink
    Oct 28 '18 at 17:51
-1
$\begingroup$

The problem in your case (as I thought previously) is the sigmoid activation function. It suffers from many problems. Out of that your performance decrease is likely due to two reasons:

NOTE: The link provided for 'Vanishing Gradient' explains beautifully why increasing layers make your network more susceptible to saturation of learning.

The vanishing gradient problem makes sure your Neural Neyt is trapped in a non optimal solution. While the high learning rate ensures that you get trapped in the non optimal solution. In short the high learning rate after a few oscillations will push your network to saturation.

Solution:

  • Best solution is to use the ReLu activation function, with maybe the last layer as sigmoid.
  • Use an adaptive optimizer like AdaGrad, Adam or RMSProp.
  • Decrease the learning rate to $10^-6$ to $10^-7$ but to compensate increase the number of epochs to $10^6$ to $10^7$.
Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ Has nothing to do with this, it's the size of the network compared to the amount of data available. $\endgroup$
    – Matthieu Brucher
    Oct 28 '18 at 13:41
  • $\begingroup$ @MatthieuBrucher what exactly do you mean by size? $\endgroup$
    – DuttaA
    Oct 28 '18 at 13:43
  • $\begingroup$ The size of the network (number of layers + number of nodes per layers). $\endgroup$
    – Matthieu Brucher
    Oct 28 '18 at 13:43
  • $\begingroup$ @MatthieuBrucher yes adding a layer makes it more prone to less learning via vanishing gradient, check the link of vanishing gradient...i did not add it my answer because the answer given was great, however I will indicate it in my answer $\endgroup$
    – DuttaA
    Oct 28 '18 at 13:44
  • $\begingroup$ I know what a vanishing gradient is... You don't know what OP uses for the training, and the number of epochs is a proof that you didn't read the question. $\endgroup$
    – Matthieu Brucher
    Oct 28 '18 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.