I wanted to recreate the model mentioned in this paper:https://arxiv.org/pdf/1610.09204v1.pdf . I am using keras with tensorflow backend, and a gtx 1050ti.

I am an ML beginner, and thought this would be a good way to get a hands on feel for things. However, My model is not converging(loss is same as first epoch). This is what I read from that paper:

The first convolutional layer re- ceives an input of 56px by 56px images with RGB channels. It uses 32 filters of size 5×5×3, stride 1 and then sampled with max pooling of size 2 × 2, stride 1. The second convolutional layer has 64 filters of size 5×5×32, stride 1 and a max pooling of size 2 × 2, stride 1. The results of the second max pooling provide the first fully-connected layer with a vector of length 12,544 (14 × 14 × 64) which are used by 512 neurons. The final fully-connected output layer uses a 20-wide softmax [21] which represents the probability of each respective 20 class labels. This architecture is similar to the LeNet model [3], but with using rectified linear unit (ReLU) [22] activation functions instead of sigmoid activation functions. We also use dropout [23], a technique to prevent overfitting, with a keep probability of 0.5 for the fully-connected layers.

and my code is:

model = Sequential()
model.add(Convolution2D(32, 5, 5, border_mode='same', input_shape=(70,52, 3))) model.add(Activation("relu")) model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Convolution2D(64, 5, 5, border_mode='same')) model.add(Activation("relu")) model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Flatten()) model.add(Dense(output_dim=512)) model.add(Activation("relu"))
model.add(Activation("softmax")) model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(x_train, y_train, nb_epoch=70, batch_size=500,verbose=1)

the full code can be found here : https://gist.github.com/harveyslash/5c98f9fdab0d53a2a48f477a52d8588d I have scrapped the data from goodreads Help appreciated !


I forgot to actually ask what i wanted. Since its my first experiment, i would like to ask what are some things that I should do to make my model converge.


You may try Stochastic Gradient Descent optimizer with a learning rate decay and nesterov momentum. You can also try a different batch_size. Also you are missing drop out layers between the fully connected layers which the authors used.


# flatten the conv layers

# fully connected 512

# fully connected output layer

# compile

# train
model.fit(x_train, y_train, 

In my experience this usually helps a lot. If you get this setting to converge, you may try rmsprop and adam.

  • $\begingroup$ Adding the extra ReLU after the flatten is doing nothing. The layer being flattened already has had that nonlinearity applied and relu( relu(x) ) == relu(x) $\endgroup$ – Neil Slater Nov 25 '16 at 9:29

There are many things that can go wrong. But the first thing I would check is the learning rate. Adam can be initialized with learning rate which isn't shown in your code. You could probably try setting a lower learning rate, say, 10e-4, first.


Not the answer you're looking for? Browse other questions tagged or ask your own question.