I did a neural network in c++ to recognize handwritten digits using the MNIST dataset without any neural network pre-existing libraries. My network has 784 inputs neuron (the pixel of the image), 100 neuron in the single hidden layer and 10 output neurons. I also have 1 bias neuron per layer. My activation function is the sigmoid function (top get ouputs between 0 and 1). My cost function is the root mean squared error. i have a learning rate 0.1 and a gradient descent momentum of 0.3.

I get an accuracy of about 90% on the MNIST training set. However on the testing set I only get a 30% accuracy. I did check the error rate during the training and I saw that it decreases for the first 120 epochs. Then it gets stuck at 0.30.

That is a plot of the cost function during all 60 000 epochs of the training :
enter image description here

This is also the cost function during the training but only the first 600 epochs : enter image description here

And this is a plot of the cost function during the validation set (10 000 epochs) : enter image description here

First I suspected an overfitting problem, so I tried stopping the learning right after the plateau appeared (at 120 epochs). It was worse : in the testing set, I only got 10% accuracy. I also tried dropout regularization which didn't work, so I don't really know if it's an overfitting related problem.

I tried using the ReLU activation function for the hidden layers instead of the sigmoid to avoid the vanishing gradient problem without success (I got down to 10% accuracy).

Then, I tried changing the learning rate, but the plateau still appeared at 120 epochs, as I thought it would at least delay the "plateau" apparition. I also tried disabling the gradient descent momentum without any success.

I thought my cost function was stuck in a local minimum (explaining the plateau at 0.3), so I tried decreasing the learning rate as the network WAS training (starting at 0.5 and decreasing it by a fraction of number of epochs), hoping it would make the cost function "unstuck" from that local minumum but it didn't.

I tried changing the size of the hidden layer, even adding one more, but it didn't change anything. There is still a "plateau" and it still starts at 120 epochs.

What can I do to solve this problem ?

  • $\begingroup$ 1. instead of guessing, do you have an actual plot of loss vs. epoch? 2. where is the validation set? Please plot the loss of validation set on top of train set too. $\endgroup$
    – lpounng
    Jul 14, 2023 at 7:06
  • $\begingroup$ Thanks for the answer ! I did add the plots to the question and while doing the plots I actually noticed that the cost function gets stuck at 120 epochs (and not 1200). $\endgroup$
    – kripi
    Jul 14, 2023 at 17:07
  • $\begingroup$ Please plot the first 200 epochs of training and validation cost on the same plot (with different color). It is meaningless to plot 60K epochs. Also, make a separate plot of training and validation accuracy vs. epochs. $\endgroup$
    – lpounng
    Jul 18, 2023 at 1:59


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.