0
$\begingroup$

I am using the ResNet50 pretrained model to train my images using TensorFlow. I have 70k images and upgraded to Google Colab Pro, but still I am facing a memory error. So how many images I can train in Google Colab? And how much RAM is needed for 70k images?

This is how I labeled and loaded images from the drive.

labels = []

imagePaths_generater = paths.list_images(Config.DATASET_PATH)
imagePaths = []
for item in imagePaths_generater:
  imagePaths.append(item)

for imagePath in imagePaths:
  label = imagePath.split(os.path.sep)[-2].split("_")
  #print(label)
  image = load_img(imagePath, target_size=(224, 224))
  image = img_to_array(image)
  image = preprocess_input(image)
  data.append(image)
  labels.append(label)
 

data = np.array(data, dtype="float")
labels = np.array(labels)```
Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Are you loading them in memory all at once? $\endgroup$
    – noe
    Dec 28, 2020 at 14:55
  • $\begingroup$ yes. i am loading at once. $\endgroup$
    – Bala venkatesh
    Dec 29, 2020 at 8:01
  • 1
    $\begingroup$ You should train in minibatches, and only load one minibatch of images at a time. $\endgroup$
    – noe
    Dec 29, 2020 at 9:03
  • 1
    $\begingroup$ I have updated my question with my image load function. how can I load as a minibatch? $\endgroup$
    – Bala venkatesh
    Dec 29, 2020 at 9:31
  • $\begingroup$ I added an answer with the information about this. $\endgroup$
    – noe
    Dec 29, 2020 at 10:16

1 Answer 1

Reset to default
4
$\begingroup$

The real problem is that you should not try to fit all your images in memory. Instead, you should small groups of images, normally called "minibatches", which can fit in the GPU/CPU memory.

For that, tensorflow offers the function tf.keras.preprocessing.image_dataset_from_directory that loads images from a directory. I suggest you take a look at Tensorflow's tutorial for image loading, which guides you to the image loading process, as well as the training of a model with those images.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks. The problem i am facing is how can i use this method for multi class classification. as you see my code i used label = imagePath.split(os.path.sep)[-2].split("_") split function to get label. how can i label using image_dataset_from_directory method. can you please share your idea on this. $\endgroup$
    – Bala venkatesh
    Dec 29, 2020 at 13:44
  • $\begingroup$ This is how i named my folder name Apple_AppleScab so apple is one class and applescab is another label so how can i label using the image_dataset_from_directory method. $\endgroup$
    – Bala venkatesh
    Dec 29, 2020 at 13:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.