1
$\begingroup$

I am trying to train an autoencoder CNN on Google Colab using keras. I have mounted my Google Drive which contains all training data. The training uses six workers and the data is loaded by the following custom generator class:

class ImageDataGenerator(keras.utils.Sequence):

    def __init__(self, filepaths, batch_size, shape, shuffle=True):
        self.shape = shape
        self.batch_size = batch_size
        self.filepaths = filepaths
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        return int(np.ceil(len(self.filepaths) / self.batch_size))

    def __getitem__(self, index):
        from_index = index * self.batch_size
        to_index = min(len(self.filepaths), (index + 1) * self.batch_size)

        filepaths_temp = self.filepaths[from_index : to_index]

        images = load_images(filepaths_temp) / 255.0

        return images, images

    def on_epoch_end(self):
        if self.shuffle:
            np.random.shuffle(self.filepaths)

My problem: Sometimes when I start training I get this:

Epoch 1/100
/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py:610: UserWarning: The input 1303 could not be retrieved. It could be because a worker has died.
  UserWarning)
1599/1599 [==============================] - 2461s 2s/step - loss: 0.0239 - val_loss: 0.0251

Epoch 00001: val_loss improved from inf to 0.02512, saving model to /content/drive/...
Epoch 2/100

Apart from the warning epoch 1 seems to finish successfully. But after the last line is printed, nothing happens anymore. Epoch 2 never starts.

I suspect the mounted Google Drive filesystem is the problem, because I had problems with Google Drive timeouts in the past. My Question is: Is there something I did wrong? Did one of the six workers die? Is there something I can do to restore the workers after the first epoch? After all, it seemed to work fine for the first epoch.

$\endgroup$
  • $\begingroup$ I have the same problem appearing from time to time when fitting in multiprocessing mode. It highly depends on the queue size I use with respect to the number of workers I use. It's super annoying because I didn't identify a pattern yet of this warning (and then bug appearing) $\endgroup$ – Zaccharie Ramzi Sep 27 '19 at 15:41
  • $\begingroup$ @ZaccharieRamzi One thing that seems to help in my case is to copy all training data from the mounted Drive file system to the Google Colab file system. I will add that as a solution if I don't encounter the problem anymore in the future. $\endgroup$ – theblackips Oct 3 '19 at 11:04
  • $\begingroup$ I am not using Colab. However my data is indeed on a hard drive not on my system. I have too much of it however, so it's not possible for me to copy it on the system. One quick-fix that works for me is to have a low max_queue_size. $\endgroup$ – Zaccharie Ramzi Oct 3 '19 at 11:42
  • $\begingroup$ Make sure the drive which contains your data is stable. Mounting google drive is not stable. $\endgroup$ – mhndlsz Oct 15 '19 at 17:56
0
$\begingroup$

The solution for me was to copy all training data from the mounted Google Drive file system to the Google Colab file system. I didn't realize before that the mounted Drive file systems has a comparatively bad performance that leads to this error.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.