0
$\begingroup$

Trying to wrap my head around where Tensorflow starts to use a lot of memory.

If I have a batch_size=10 and my samples are 150MB each, then that’s only ~1GB at a time or less if the samples are processed 1 at a time.

Is it more about the accruing size of the neural net itself as I go through more epochs? If I try to train against 10,000 of these 150MB samples, am I more likely to run out of memory toward the end of the training?

$\endgroup$
1
$\begingroup$

Assuming your samples have a fixed size and you use constant batch size, you should never run out of memory - that is: as soon as you have managed to train a full batch (forward and backward propagation complete) you should be fine to assume it will train until completion!


Extra info / tips

The total memory requirement will be a product of the size of the model (number of parameters) along with the batch size. Each single sample that gets sent through the network will effectively influence each of the $N$ weights. A batch of size $B$ will mean that there needs to be roughly $N \times B$ memory available.

If you model is already fixed, you can start with a large batch size, decreasing it slowly until you no longer receive an outOfMemory error. You could also try computing exactly what your memory requirement are, but there are some other factors that might make that a little difficult, so it is faster to try the first approach.

If you are using a GPU, you ca look at the output of the terminal command nvidia-smi, you can see the available memory of the GPUs. You will notice it essentially becomes all used as soon as training begins. This is because Tensorflow, by default, will occupy all available memory. There are ways around that, so search for allow_growth in the Tensorflow docs for relevant information and configuration tactics.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.