# cifar10 official keras example not giving expected accuracy, using sigmoid seems better than relu

In the official Keras example cifar10 there is the following code to train a CNN using keras10. When I tried it, my neural net would not learn at all, I always get around a 10% acuracy, which is basicaly random guessing. If I change in the first dense layer the activation from "reluat " to "sigmoid", then I get much better results. In fact, I thought that before softmax one had to have a sigmoid, not a relu. However, this example appears all over the internet, so I am sure it is correct. What am I missing?

(P.S. I am training using a CPU, but that should not affect performance, just take longer, also, I stopped at around 20 epochs, but still with the sigmoid I got almost 70%, so that should not be the problem either)

'''Train a simple deep CNN on the CIFAR10 small images dataset.
GPU run command:
THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python cifar10_cnn.py
It gets down to 0.65 test logloss in 25 epochs, and down to 0.55 after 50 epochs.
(it's still underfitting at that point, though).
Note: the data was pickled with Python 2, and some encoding issues might prevent you
save it in a different format, load it in Python 3 and repickle it.
'''

from __future__ import print_function
from keras.datasets import cifar10
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Convolution2D, MaxPooling2D
from keras.optimizers import SGD
from keras.utils import np_utils

batch_size = 32
nb_classes = 10
nb_epoch = 200
data_augmentation = False

# input image dimensions
img_rows, img_cols = 32, 32
# the CIFAR10 images are RGB
img_channels = 3

# the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
print('X_train shape:', X_train.shape)
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)

model = Sequential()

input_shape=(img_channels, img_rows, img_cols)))

# let's train the model using SGD + momentum (how original).
sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy',
optimizer=sgd,
metrics=['accuracy'])

X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255

if not data_augmentation:
print('Not using data augmentation.')
model.fit(X_train, Y_train,
batch_size=batch_size,
nb_epoch=nb_epoch,
validation_data=(X_test, Y_test),
shuffle=True)
else:
print('Using real-time data augmentation.')

# this will do preprocessing and realtime data augmentation
datagen = ImageDataGenerator(
featurewise_center=False,  # set input mean to 0 over the dataset
samplewise_center=False,  # set each sample mean to 0
featurewise_std_normalization=False,  # divide inputs by std of the dataset
samplewise_std_normalization=False,  # divide each input by its std
zca_whitening=False,  # apply ZCA whitening
rotation_range=0,  # randomly rotate images in the range (degrees, 0 to 180)
width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
horizontal_flip=True,  # randomly flip images
vertical_flip=False)  # randomly flip images

# compute quantities required for featurewise normalization
# (std, mean, and principal components if ZCA whitening is applied)
datagen.fit(X_train)

# fit the model on the batches generated by datagen.flow()
model.fit_generator(datagen.flow(X_train, Y_train,
batch_size=batch_size),
samples_per_epoch=X_train.shape[0],
nb_epoch=nb_epoch,
validation_data=(X_test, Y_test))

• I would suggest doing the following changes: 1) Add one more dense layer with like 50-100 units after the dense layer having 512 units in the code. 2) Increase your dropout from 0.25 to atleast 0.5 3) Lower your learning rate to 0.001 or 0.0001. 4) Try another optimizer like adam 5) Increase the depth of your network by adding more COnv layers with more filters Jan 28 '17 at 9:06

In this example the learning rate of Stochastic Gradient Descent mentioned is too large and reducing it to 0.001 will solve the issue (but the learning will be slow).

The difference between sigmoid and relu arises because relu has a slope of 1 (constant) while sigmoid gets saturated and therefore can have a much smaller gradient. Therefore even with the high learning rate sigmoid survives but relu keeps fluctuating around 10%.

There are several changes between current version of cifar10_cnn.py and the code in your question. However, both version is not that bad (10% accuracy) in my environment.

X_train shape: (50000, 32, 32, 3)

50000 train samples

10000 test samples

Not using data augmentation.

Train on 50000 samples, validate on 10000 samples

Epoch 1/200

50000/50000 [==============================] - 21s - loss: 1.7470 - acc: 0.3528 - val_loss: 1.3405 - val_acc: 0.5222

...

Epoch 20/200

50000/50000 [==============================] - 20s - loss: 0.6308 - acc: 0.7800 - val_loss: 0.7183 - val_acc: 0.7588

Result of cifar10_cnn.py:

X_train shape: (50000, 32, 32, 3)

50000 train samples

10000 test samples

Using real-time data augmentation.

Epoch 1/200

50000/50000 [==============================] - 26s - loss: 1.5978 - acc: 0.4226 - val_loss: 1.1876 - val_acc: 0.5712

...

Epoch 20/200

50000/50000 [==============================] - 24s - loss: 1.7479 - acc: 0.4229 - val_loss: 1.6603 - val_acc: 0.4292