1
$\begingroup$

Currently, I'm loading my images for the training model with TensorFlow, like this:

for index, datum in data.iterrows():
  sat_name = directory_path+datum['sat_image_path']
  sat_img = plt.imread(sat_name)
  sat = cv.resize(sat_img,(SIZE,SIZE))
  tf_data['sat'].append(sat)
  print("Progress: ",(100*index)/data.size,"%")
  clear_output(wait=True)

I know this is not the best way to do it. But this is what I got. This data is DataFrame, swhich holds all the image-ids and their relative paths.

And this seems, pretty slow, it takes like 50 minutes in Colab, to load 12k images.

And when feeding to my model, I do something like this:

unet.fit(
      np.array(tf_data['sat']), 
      ...

Can anyone suggest to me a better way to do it, more faster and elegant and surely updated way?

$\endgroup$
2
  • 1
    $\begingroup$ Instead of loading all images into memory before fitting your model, you could write a custom data generator that loads the images as they are needed in the training process. In addition, you could try to load images using cv2.imread instead of plt.imread which may be a bit faster. $\endgroup$
    – Oxbowerce
    Oct 26 at 12:58
  • $\begingroup$ @Oxbowerce that made it faster, I guess you should post it as an answer... Dramatically reduced time to one-fourth... $\endgroup$ Oct 29 at 6:37
1
$\begingroup$

Instead of loading all images into memory before fitting your model, you could write a custom data generator that loads the images as they are needed in the training process. In addition, you could try to load images using cv2.imread instead of plt.imread which may be a bit faster.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.