I am beginning with deep learning. This is an implementation of a simple neural network with just 1 hidden layer on MNIST dataset. Why is it that the loss doesn't change at all after any epoch? It clearly means that it is not learning at all. The accuracy is approx. 11% that is like random guessing. But should it be so less?
I have used Adam optimizer and cross_entropy loss.

input_nodes = 784
hl1_nodes = 64
output_nodes = 1

from keras.datasets import mnist
(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train_reshaped = X_train.reshape(X_train.shape[0],784)

model = Sequential()
model.add(Dense(hl1_nodes, activation='relu', input_shape=(input_nodes,)))
model.add(Dense(output_nodes, activation = 'sigmoid'))

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

history = model.fit(x=X_train_reshaped, y=y_train, validation_split=0.33, verbose=1,epochs=10)

Train on 40199 samples, validate on 19801 samples
Epoch 1/10
40199/40199 [==============================] - 4s 87us/step - loss: -55.0254 - acc: 0.1142 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 2/10
40199/40199 [==============================] - 3s 76us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 3/10
40199/40199 [==============================] - 3s 74us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 4/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 5/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 6/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 7/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 8/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 9/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 10/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088

What am I missing? Edit: It is same upto the 4th digit even after 90th epoch.


What am I missing?

  • Incorrect architecture for the classification task. You have a single binary output, trained using binary_crossentropy, so the NN can only classify something as in a class (label 1) or not (label 0). Instead, you most likely want 10 outputs using softmax activation instead of sigmoid, and categorical_crossentropy as the loss, so that you can classify which digit is most likely given an input.

  • Incomplete processing of MNIST raw data. The input pixel values in X_train range from 0 to 255, and this will cause numeric problems for a NN. The target labels in y_train are the digit value (0,1,2,3,4,5,6,7,8,9), whilst for classification you will need to turn that into binary classes - usually a one-hot coding e.g. the label 3 becomes a vector [0,0,0,1,0,0,0,0,0,0].

    • Scale the inputs - a quick fix might be X_train = X_train/ 255 and X_test = X_test/ 255
    • One-hot code the labels. A quick fix might be y_train = keras.utils.to_categorical(y_train)

I made those changes to your code and got this after 10 epochs:

val_loss: 0.1194 - val_acc: 0.9678

There are a thousand tricks you can use to improve accuracy on MNIST. I am indebted to the Yassine Ghouzam Kaggle Kernel for most of these ideas:

  1. Normalize the data. This allows the optimization to run a bit faster.
  2. Use the Conv2D layers in keras, with MaxPool2D every so often. The Ghouzam kernel uses Conv2D, Conv2D, MaxPool, Dropout, Conv2D, Conv2D, MaxPool2D, Dropout, Flatten, Dense, Dropout, and finally a Dense at the end. All activation functions, where applicable, are relu, except the output layer which is softmax.
  3. The other key to getting a high accuracy is data generation. This works particularly well on MNIST because it's easy to tweak an image slightly without changing the label inadvertently. You can use the ImageDataGenerator from keras to do this.

I ran the Ghouzam kernel for 60 epochs, which took quite a while on my under-powered hardware, but I got $99.6\%$ accuracy when I submitted to the Kaggle competition.


You have used extremely naive approach one could follow to predict MNIST dataset.

The problem seems to be with the final output node that uses 'sigmoid' activation function.

Sigmoid activation function predicts values between zero and one, and is the wrong choice of activation function for predicting numbers ranging from 0 to 9 (To verify this, try to append the correct predictions into a list and print it.).

One approach you can use is to one-hot encode the labels and then feed the labels to the network. This will certainly solve your problem.

After you are done with this, you can try a number of ways to improve your accuracy for the model.

  • Use simple feature extraction algorithm such as HOG
  • Use convolution network. You can learn about it here.
  • Apply data augmentation. Refer here.
  • Build a deeper model.
  • Use various kinds of network architectures.
  • Use transfer learning. ...

There could be immense number of solutions to this that have been tried and tested. Just try different approaches yourself, read different blogs and understand other people's approach for the same and your model will definitely improve.


I think your architecture here is too naive. However, I was able to reproduce same low accuracy with a good CNN architecture i.e. my model was using softmax instead of sigmoid for MNIST dataset.

model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape= (28,28,1)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.Conv2D(64, (3,3), activation='relu'))

#Flaten the 3D conv-net to 1D for the Dense layers

model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10, activation='softmax'))`

Even then, I was getting a very low accuracy. However, region for my low accuracy was not architecture of CNN. Rather the shape of X i.e. input array. In your case shape of X_train and X_test.

X_train = X_train.reshape(-1,28,28,1)
X_train = X_train.reshape(-1,28,28,1)

Upon, reshaping the train and test array. My accuracy shot to 99+. Please try reshaping your input array along with better architecture.

  • $\begingroup$ Thanks @Stephen for the edit. $\endgroup$ – DataFramed Sep 10 '19 at 7:34
  • $\begingroup$ Can you please help me to learn more about the shape and reshapes I struggle with this a lot. $\endgroup$ – silentsudo Mar 12 '20 at 7:22
  • 1
    $\begingroup$ Hi @silentsudo, reshape function helps change dimension e.g. in this case we were provided with 1D array containing 784 elements ( pixel value). This 1D array contains pixel value of digital image of size 28 x 28 in greyscale. Therefore, we gave matching argument to the reshape(-1,28,28,1) function. This converts the 1D array into 2D pixel array which the above CNN model uses. Here, 2nd and 3rd arguments of reshape() are image height and width, while the last argument is set to 1 for grey-scale. Last argument needs to be 3 if image is color representing RGB pixel values. $\endgroup$ – DataFramed Mar 12 '20 at 7:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.