Setting batch size: training requires twice as much memory as validating

Question

I am using Keras with a Tensorflow backend to train an Image Classification model on a GPU. I have read somewhere that training uses roughly twice (both forward and back props) the GPU memory of validating, so therefore the training batch size should be the half of the validation batch size.

However, on many blogs and tutorials, I see that people use the same batch size for training and validating.

Is it true that training uses twice the GPU memory, because of the forward and backward pass, or is this false?

Answer 1

Aside: Tensorflow-GPU currently grabs 100% of GPU memory by design -- so that may may change the challenge a bit.

To add to @hssay on point answer -- with images, it is helpful to look at the entire pipeline. Optimize the input pipeline, then you will use less memory. May want to start here: TensorFlow

Test, test test. You may find that your results do not really change much with smaller image files (less resolution, color, etc). In some cases, results may even get better (the overtraining/undertraining balance).

[this is really should be a comment, but my rep does not transfer here :( ]

Answer 2

Aside: The terminology of Keras/Tensorflow is (probably) little non-standard when they talk about validation sets. In conventional ML terms, validation set is used to find hyperparameters where as test set is used to quantify accuracy of selected model. So in below explanation, I'm assuming that you are talking about training and test set.

Every parameter of the network (weight) influences outcome and hence the cost function. To tune the weight, you need derivative of cost function with respect to weights. So during training, you have to store gradient values and weights both. During inference stage, you only need weights. So that is where the memory difference comes from.