# My first machine learning experiment , model not converging , tips? [closed]

I wanted to recreate the model mentioned in this paper:https://arxiv.org/pdf/1610.09204v1.pdf . I am using keras with tensorflow backend, and a gtx 1050ti.

I am an ML beginner, and thought this would be a good way to get a hands on feel for things. However, My model is not converging(loss is same as first epoch). This is what I read from that paper:

The first convolutional layer re- ceives an input of 56px by 56px images with RGB channels. It uses 32 filters of size 5×5×3, stride 1 and then sampled with max pooling of size 2 × 2, stride 1. The second convolutional layer has 64 filters of size 5×5×32, stride 1 and a max pooling of size 2 × 2, stride 1. The results of the second max pooling provide the first fully-connected layer with a vector of length 12,544 (14 × 14 × 64) which are used by 512 neurons. The final fully-connected output layer uses a 20-wide softmax [21] which represents the probability of each respective 20 class labels. This architecture is similar to the LeNet model [3], but with using rectified linear unit (ReLU) [22] activation functions instead of sigmoid activation functions. We also use dropout [23], a technique to prevent overfitting, with a keep probability of 0.5 for the fully-connected layers.

and my code is:

model = Sequential()
model.fit(x_train, y_train, nb_epoch=70, batch_size=500,verbose=1)


the full code can be found here : https://gist.github.com/harveyslash/5c98f9fdab0d53a2a48f477a52d8588d I have scrapped the data from goodreads Help appreciated !

EDIT

I forgot to actually ask what i wanted. Since its my first experiment, i would like to ask what are some things that I should do to make my model converge.

You may try Stochastic Gradient Descent optimizer with a learning rate decay and nesterov momentum. You can also try a different batch_size. Also you are missing drop out layers between the fully connected layers which the authors used.

Try

...
# flatten the conv layers

# fully connected 512

# fully connected output layer

# compile
model.compile(loss='categorical_crossentropy',
optimizer=SGD(lr=10e-4,
decay=10e-6,
momentum=0.99,
nesterov=True),
metrics=['accuracy'])

# train
model.fit(x_train, y_train,
nb_epoch=70,
batch_size=64,
verbose=1)


In my experience this usually helps a lot. If you get this setting to converge, you may try rmsprop and adam.

• Adding the extra ReLU after the flatten is doing nothing. The layer being flattened already has had that nonlinearity applied and relu( relu(x) ) == relu(x) – Neil Slater Nov 25 '16 at 9:29

There are many things that can go wrong. But the first thing I would check is the learning rate. Adam can be initialized with learning rate which isn't shown in your code. You could probably try setting a lower learning rate, say, 10e-4, first.