PLEASE NOTE: I am not trying to improve on the following example. I know you can get over 99% accuracy. The whole code is in the question. When I tried this simple code I get around 95% accuracy, if I simply change the activation function from sigmoid to relu, it drops to less than 50%. Is there a theoretical reason why this happens?
I have found the following example online:
from keras.datasets import mnist from keras.models import Sequential from keras.layers.core import Dense, Activation from keras.utils import np_utils (X_train, Y_train), (X_test, Y_test) = mnist.load_data() X_train = X_train.reshape(60000, 784) X_test = X_test.reshape(10000, 784) Y_train = np_utils.to_categorical(Y_train, classes) Y_test = np_utils.to_categorical(Y_test, classes) batch_size = 100 epochs = 15 model = Sequential() model.add(Dense(100, input_dim=784)) model.add(Activation('sigmoid')) model.add(Dense(10)) model.add(Activation('softmax')) model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer='sgd') model.fit(X_train, Y_train, batch_size=batch_size, epochs=epochs, verbose=1) score = model.evaluate(X_test, Y_test, verbose=1) print('Test accuracy:', score)
This gives about 95% accuracy, but if I change the sigmoid with the ReLU, I get less than 50% accuracy. Why is that?