Maximum number of images to train a convolutional neural network

Question

I just wondered if there is a technical limit on the number of images to train a neural network.

I want to work with extremely high numbers of images, around 1,000,000 to 10,000,000 images. Is there a cap because of the graphics card memory? I tried to do some research on that issue, but I mostly find questions about how many images you need to get decent results for a convolutional neural network.

Thanks and best regards,

beinando

Answer 1

Theoretically speaking, there is no limit on how many data points you can use to train any neural network. This is true because neural networks that can be trained incrementally and do not need to "see" the entire dataset.

Obviously, it helps if your machine is able to store at least a single image and the network in memory.

To train on a large number of images, you will likely need to use batches. This means that each iteration happens on a subset of your data, which can be as small as a single image.

Answer 2

There is not any CAP by any Framework Or technology.

This is how components will be impacted -

Disc space
This will be directly impacted by the total count and size of each image

Memory
This is dependent on your layer's depth, batch size, Kernel count and size of the individual image. Please check SE/Internet on "RAM needed during CNN learning"
It means it will not change much whether image count is 1000 Or 100K if above-mentioned parameters are fixed

Training time
This will depend on your hardware and the level of parallelism but you can't do more than the batch size as backdrop need to reconcile gradients after every batch

What you must ponder
Does your images have so much of variation. Otherwise learning will be stopped after a certain image count. In case it has got variance across all images, then learning will be very slow. What it means is that Training time is the only thing you may worry about