How to load numerous files from google drive into colab

Question

I am trying to load in 30k images (600mb) from Google drive into Google Colaboratory to further process them with Keras/PyTorch.

Therefore I have first mounted my Google drive using:

from google.colab import drive
drive.mount('/content/gdrive')

Next I have unzipped the image file using:

!unzip -uq "/content/gdrive/My Drive/path.zip" -d "/content/gdrive/My Drive/path/"

Counting how many files are located in the directory using:

len(os.listdir(path-to-train-images))

I only find 13k images (whereas I should find 30k). According to the output of unzip, the files appear to be unzipped correctly.

Also, I found there are some issues with loading in many files from a google directory: https://github.com/googlecolab/colabtools/issues/510.

Does anyone know where I am going wrong? Or whether there is a workaround?

Answer 1

One possible option would be operate directly on the zip files using zipfile.ZipFile.

Counting the number of items in a zip file:

from contextlib import closing
from zipfile import ZipFile

with closing("/content/gdrive/My Drive/path.zip") as zip_file:
    count = len(zip_file.infolist())

Answer 2

There could be a few reasons why you are only seeing 13k images after unzipping the file. One possibility is that the zip file only contains 13k images, and not 30k as you expected. Another possibility is that there could be some errors during the unzipping process, which could result in some of the images not being extracted properly.

As for the issue with loading many files from a Google directory, it is possible that there is a limit on the number of files that can be loaded from a Google directory at once. One workaround for this issue is to use the glob module to load the files in batches, instead of loading them all at once. For example:

import glob

# Set the path to the directory containing the images
image_dir = '/content/gdrive/My Drive/path/'

# Use glob to get a list of all image files in the directory
image_files = glob.glob(image_dir + '*.jpg')

# Load the first batch of images (e.g. the first 1000 images)
images = []
for image_file in image_files[:1000]:
  image = cv2.imread(image_file)
  images.append(image)

You can then repeat this process for the remaining images in the directory, by incrementing the index of the image file list (e.g. image_files[1000:2000], image_files[2000:3000], etc.).

I hope this helps. Let me know if you have any other questions.