I am using the ResNet50 pretrained model to train my images using TensorFlow. I have 70k images and upgraded to Google Colab Pro, but still I am facing a memory error. So how many images I can train in Google Colab? And how much RAM is needed for 70k images?

This is how I labeled and loaded images from the drive.

labels = []

imagePaths_generater = paths.list_images(Config.DATASET_PATH)
imagePaths = []
for item in imagePaths_generater:

for imagePath in imagePaths:
  label = imagePath.split(os.path.sep)[-2].split("_")
  image = load_img(imagePath, target_size=(224, 224))
  image = img_to_array(image)
  image = preprocess_input(image)

data = np.array(data, dtype="float")
labels = np.array(labels)```
  • 1
    $\begingroup$ Are you loading them in memory all at once? $\endgroup$
    – noe
    Dec 28 '20 at 14:55
  • $\begingroup$ yes. i am loading at once. $\endgroup$ Dec 29 '20 at 8:01
  • 1
    $\begingroup$ You should train in minibatches, and only load one minibatch of images at a time. $\endgroup$
    – noe
    Dec 29 '20 at 9:03
  • 1
    $\begingroup$ I have updated my question with my image load function. how can I load as a minibatch? $\endgroup$ Dec 29 '20 at 9:31
  • $\begingroup$ I added an answer with the information about this. $\endgroup$
    – noe
    Dec 29 '20 at 10:16

The real problem is that you should not try to fit all your images in memory. Instead, you should small groups of images, normally called "minibatches", which can fit in the GPU/CPU memory.

For that, tensorflow offers the function tf.keras.preprocessing.image_dataset_from_directory that loads images from a directory. I suggest you take a look at Tensorflow's tutorial for image loading, which guides you to the image loading process, as well as the training of a model with those images.

  • $\begingroup$ Thanks. The problem i am facing is how can i use this method for multi class classification. as you see my code i used label = imagePath.split(os.path.sep)[-2].split("_") split function to get label. how can i label using image_dataset_from_directory method. can you please share your idea on this. $\endgroup$ Dec 29 '20 at 13:44
  • $\begingroup$ This is how i named my folder name Apple_AppleScab so apple is one class and applescab is another label so how can i label using the image_dataset_from_directory method. $\endgroup$ Dec 29 '20 at 13:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.