Keras intuition/guidelines for setting epochs and batch size

Question

I'm using Python with Keras to make a convolutional neural network (CNN) for an image classifier. I took about 50 images of documents and 150 images of non-documents for training. I shrunk the resolution, using scipy.misc.imresize(small, (32,32)) to standardize the images, when looking through these pixelated images I thought that I could still tell the difference between photos and documents, so I figured that the ML algorithm should be able to as well.

My question is in regards to the number of epochs and batch size. I'm currently using model.fit(X_train_temp, y_train_temp, epochs=N_epochs, batch_size=batch_size, verbose=verbose) . I've seen this question regarding the topic here:

How to set batch_size, steps_per epoch and validation steps

But the answer was very definition-based; I'm looking for intuition. I would like to know if there are general guidelines as to what values to set the number of epochs and batch size to for a given problem. I performed a crude parameter sweep across the number of epochs and batch size. Here is the CNN model:

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), input_shape=input_shape, activation='sigmoid'))
model.add(Conv2D(32, kernel_size=(3, 3), input_shape=input_shape, activation='sigmoid'))
model.add(Flatten())
model.add(Dense(num_classes, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

And here are the results:

Here are some observations that I've made so far. The circled region seems to be pretty good for training since high accuracy is achieved relatively early on and it does not seem to oscillate much as further epochs pass. It seems that batch_size = 1 is generally a bad idea since training does not seem to improve the model. Additionally, it seems that batch_size >~ N_epochs seems to be desirable.

Question 1: is batch_size = 1 generally a bad? Or is this just for my particular parameter sweep?

Question 2: Is it generally the case that N_epochs >~ batch_size is recommended?

Question 3: Can anyone speak to the robustness of the generalizations in questions 2?

Thank you.

Answer 1

Actually it's controversial and it can differ from one problem to another. the best thing to do is to search the hyperparamteres space for an optimal result.

• For Question 1

Here I quot the father of CNN (Yann Lecun) : " Training with large minibatches is bad for your health. More importantly, it's bad for your test error. Friends dont let friends use minibatches larger than 32.

Let's face it: the only people have switched to minibatch sizes larger than one since 2012 is because GPUs are inefficient for batch sizes smaller than 32. That's a terrible reason. It just means our hardware sucks.

What's worse is that the easiest way to parallelize training is to make the minibatch even larger and distribute it across multiple GPUs and multiple nodes. Minibatch sizes over 1024 aren't just bad for your health. They cause brain tumors. They learn quickly, but the wrong thing. ...

Update2: I'm pointing to this particular paper as a way to make a general point. But I'm not necessarily endorsing the particular results nor the methodology in this paper (some folks have pointed out a few flaws in the methodology). Still, the main point is that the only reason to use a batch size larger than 1 is the limitations of the limitations of the hardware at our disposal."

In this paper a value for batches between 2 and 32 is recommended

• For Questions 2 & 3:

Usually an early stopping technique is used by setting the number of epochs to a very large number and when the generalization error gets worse we just stop. please see

For more details and explanations please see

At the end I would recommend using "ReLu" activation function for the hidden layers rather than the "sigmoid". it learns much faster