Too low accuracy on MNIST dataset using a neural network

Question

I am beginning with deep learning. This is an implementation of a simple neural network with just 1 hidden layer on MNIST dataset. Why is it that the loss doesn't change at all after any epoch? It clearly means that it is not learning at all. The accuracy is approx. 11% that is like random guessing. But should it be so less?
I have used Adam optimizer and cross_entropy loss.

input_nodes = 784
hl1_nodes = 64
output_nodes = 1

from keras.datasets import mnist
(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train_reshaped = X_train.reshape(X_train.shape[0],784)

model = Sequential()
model.add(Dense(hl1_nodes, activation='relu', input_shape=(input_nodes,)))
model.add(Dense(output_nodes, activation = 'sigmoid'))

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

history = model.fit(x=X_train_reshaped, y=y_train, validation_split=0.33, verbose=1,epochs=10)

#output
Train on 40199 samples, validate on 19801 samples
Epoch 1/10
40199/40199 [==============================] - 4s 87us/step - loss: -55.0254 - acc: 0.1142 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 2/10
40199/40199 [==============================] - 3s 76us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 3/10
40199/40199 [==============================] - 3s 74us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 4/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 5/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 6/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 7/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 8/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 9/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 10/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088

What am I missing? Edit: It is same upto the 4th digit even after 90th epoch.

Answer 1

What am I missing?

• Incorrect architecture for the classification task. You have a single binary output, trained using binary_crossentropy, so the NN can only classify something as in a class (label 1) or not (label 0). Instead, you most likely want 10 outputs using softmax activation instead of sigmoid, and categorical_crossentropy as the loss, so that you can classify which digit is most likely given an input.

• Incomplete processing of MNIST raw data. The input pixel values in X_train range from 0 to 255, and this will cause numeric problems for a NN. The target labels in y_train are the digit value (0,1,2,3,4,5,6,7,8,9), whilst for classification you will need to turn that into binary classes - usually a one-hot coding e.g. the label 3 becomes a vector [0,0,0,1,0,0,0,0,0,0].

  • Scale the inputs - a quick fix might be X_train = X_train/ 255 and X_test = X_test/ 255
  • One-hot code the labels. A quick fix might be y_train = keras.utils.to_categorical(y_train)

I made those changes to your code and got this after 10 epochs:

val_loss: 0.1194 - val_acc: 0.9678

Answer 2

There are a thousand tricks you can use to improve accuracy on MNIST. I am indebted to the Yassine Ghouzam Kaggle Kernel for most of these ideas:

1. Normalize the data. This allows the optimization to run a bit faster.
2. Use the Conv2D layers in keras, with MaxPool2D every so often. The Ghouzam kernel uses Conv2D, Conv2D, MaxPool, Dropout, Conv2D, Conv2D, MaxPool2D, Dropout, Flatten, Dense, Dropout, and finally a Dense at the end. All activation functions, where applicable, are relu, except the output layer which is softmax.
3. The other key to getting a high accuracy is data generation. This works particularly well on MNIST because it's easy to tweak an image slightly without changing the label inadvertently. You can use the ImageDataGenerator from keras to do this.

I ran the Ghouzam kernel for 60 epochs, which took quite a while on my under-powered hardware, but I got $99.6\%$ accuracy when I submitted to the Kaggle competition.

Answer 3

You have used extremely naive approach one could follow to predict MNIST dataset.

The problem seems to be with the final output node that uses 'sigmoid' activation function.

Sigmoid activation function predicts values between zero and one, and is the wrong choice of activation function for predicting numbers ranging from 0 to 9 (To verify this, try to append the correct predictions into a list and print it.).

One approach you can use is to one-hot encode the labels and then feed the labels to the network. This will certainly solve your problem.

After you are done with this, you can try a number of ways to improve your accuracy for the model.

• Use simple feature extraction algorithm such as HOG
• Use convolution network. You can learn about it here.
• Apply data augmentation. Refer here.
• Build a deeper model.
• Use various kinds of network architectures.
• Use transfer learning. ...

There could be immense number of solutions to this that have been tried and tested. Just try different approaches yourself, read different blogs and understand other people's approach for the same and your model will definitely improve.

Answer 4

I think your architecture here is too naive. However, I was able to reproduce same low accuracy with a good CNN architecture i.e. my model was using softmax instead of sigmoid for MNIST dataset.

model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape= (28,28,1)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))

#Flaten the 3D conv-net to 1D for the Dense layers
model.add(layers.Flatten())

model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10, activation='softmax'))`

Even then, I was getting a very low accuracy. However, region for my low accuracy was not architecture of CNN. Rather the shape of X i.e. input array. In your case shape of X_train and X_test.

X_train = X_train.reshape(-1,28,28,1)
X_train = X_train.reshape(-1,28,28,1)

Upon, reshaping the train and test array. My accuracy shot to 99+. Please try reshaping your input array along with better architecture.