keras UserWarning: The input 1303 could not be retrieved. It could be because a worker has died

Question

I am trying to train an autoencoder CNN on Google Colab using keras. I have mounted my Google Drive which contains all training data. The training uses six workers and the data is loaded by the following custom generator class:

class ImageDataGenerator(keras.utils.Sequence):

    def __init__(self, filepaths, batch_size, shape, shuffle=True):
        self.shape = shape
        self.batch_size = batch_size
        self.filepaths = filepaths
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        return int(np.ceil(len(self.filepaths) / self.batch_size))

    def __getitem__(self, index):
        from_index = index * self.batch_size
        to_index = min(len(self.filepaths), (index + 1) * self.batch_size)

        filepaths_temp = self.filepaths[from_index : to_index]

        images = load_images(filepaths_temp) / 255.0

        return images, images

    def on_epoch_end(self):
        if self.shuffle:
            np.random.shuffle(self.filepaths)

My problem: Sometimes when I start training I get this:

Epoch 1/100
/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py:610: UserWarning: The input 1303 could not be retrieved. It could be because a worker has died.
  UserWarning)
1599/1599 [==============================] - 2461s 2s/step - loss: 0.0239 - val_loss: 0.0251

Epoch 00001: val_loss improved from inf to 0.02512, saving model to /content/drive/...
Epoch 2/100

Apart from the warning epoch 1 seems to finish successfully. But after the last line is printed, nothing happens anymore. Epoch 2 never starts.

I suspect the mounted Google Drive filesystem is the problem, because I had problems with Google Drive timeouts in the past. My Question is: Is there something I did wrong? Did one of the six workers die? Is there something I can do to restore the workers after the first epoch? After all, it seemed to work fine for the first epoch.

Answer 1

The solution for me was to copy all training data from the mounted Google Drive file system to the Google Colab file system. I didn't realize before that the mounted Drive file systems has a comparatively bad performance that leads to this error.