0.1 accuracy on MNIST fashion dataset following official Tensorflow/Keras tutorial

Question

My goal is to classify products pictures into categories such as dress, sandals, etc.

I am using the MNIST fashion dataset, following this official tutorial word-per-word: https://www.tensorflow.org/tutorials/keras/basic_classification so my code is 100% identical to what can be read there:

# TensorFlow and tf.keras
import tensorflow as tf
from tensorflow import keras

# Helper libraries
import numpy as np
import matplotlib.pyplot as plt

print(tf.__version__)

fashion_mnist = keras.datasets.fashion_mnist

(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()

class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation=tf.nn.relu),
    keras.layers.Dense(10, activation=tf.nn.softmax)
])

model.compile(optimizer=tf.train.AdamOptimizer(),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])


model.fit(train_images, train_labels, epochs=5)

test_loss, test_acc = model.evaluate(test_images, test_labels)

print('Test accuracy:', test_acc)

Problem: The resulting accuracy is always around 0.1, much lower than the tutorial's example output of 0.876.

I am obviously doing something wrong, but I can't figure out what. How to improve accuracy to something reasonable?

My output:

$ python classify-products.py 
1.10.1
Epoch 1/5
2018-09-18 13:33:46.971437: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
60000/60000 [==============================] - 3s 47us/step - loss: 13.0161 - acc: 0.1924 
Epoch 2/5
60000/60000 [==============================] - 3s 46us/step - loss: 12.8998 - acc: 0.1997
Epoch 3/5
60000/60000 [==============================] - 3s 46us/step - loss: 13.3386 - acc: 0.1724
Epoch 4/5
60000/60000 [==============================] - 3s 47us/step - loss: 12.9031 - acc: 0.1995
Epoch 5/5
60000/60000 [==============================] - 3s 47us/step - loss: 13.6666 - acc: 0.1521
10000/10000 [==============================] - 0s 26us/step
('Test accuracy:', 0.1005)

Switching to 20 epochs does not improve accuracy.

Answer 1

You haven't normalized your image dataset such as setting the pixel values between 0-1 which could help classifier converge faster.

Please do it by doing the operation below.

train_images = train_images / 255.0

test_images = test_images / 255.0

It seems you are using 50% of data for training as well as testing. try to use the data in the ratio of 7:3 for training and testing. Increase the epochs hope it will work.

Answer 2

I have the same problem, too. It is the dataset error.

I find that the training dataset loaded is the Fashion-MNIST dataset, while the test dataset is the the MNIST dataset.

So, I download the original Fashion-MNIST dataset from the official site https://github.com/zalandoresearch/fashion-mnist

Answer 3

The issue lies in the mismatch between the standard deviations of train_images vs that of the first hidden layer. The 0.05 comes from the default std of the kernel initializer of the Dense layer. The goal is to get these values closer to each other, Ideally in the same order of magnitude (around or smaller than 1.0).

train_images.std()
>> 90.0
model.layers[1].get_weights()[0].std()
>> 0.05

255 isn't a magic number. You can use 100 or 1000 too. You can also specify the desired std of the kernel initializer so that it matches std of your data.

E.g. The following will achieve the desired high accuracy in model.fit() just as well.

train_images = train_images / 100.0  # std = 1.0
train_images.std()
>> 0.9

from tensorflow.keras import initializers
ki = initializers.RandomNormal(stddev=1.0)

model = keras.models.Sequential([
  keras.layers.Flatten(input_shape=[28,28]),
  keras.layers.Dense(300, activation="relu", kernel_initializer=ki), 
  keras.layers.Dense(100, activation="relu"),
  keras.layers.Dense(10, activation="softmax")
])