# keras UserWarning: The input 1303 could not be retrieved. It could be because a worker has died

I am trying to train an autoencoder CNN on Google Colab using keras. I have mounted my Google Drive which contains all training data. The training uses six workers and the data is loaded by the following custom generator class:

class ImageDataGenerator(keras.utils.Sequence):

def __init__(self, filepaths, batch_size, shape, shuffle=True):
self.shape = shape
self.batch_size = batch_size
self.filepaths = filepaths
self.shuffle = shuffle
self.on_epoch_end()

def __len__(self):
return int(np.ceil(len(self.filepaths) / self.batch_size))

def __getitem__(self, index):
from_index = index * self.batch_size
to_index = min(len(self.filepaths), (index + 1) * self.batch_size)

filepaths_temp = self.filepaths[from_index : to_index]

return images, images

def on_epoch_end(self):
if self.shuffle:
np.random.shuffle(self.filepaths)


My problem: Sometimes when I start training I get this:

Epoch 1/100
/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py:610: UserWarning: The input 1303 could not be retrieved. It could be because a worker has died.
UserWarning)
1599/1599 [==============================] - 2461s 2s/step - loss: 0.0239 - val_loss: 0.0251

Epoch 00001: val_loss improved from inf to 0.02512, saving model to /content/drive/...
Epoch 2/100


Apart from the warning epoch 1 seems to finish successfully. But after the last line is printed, nothing happens anymore. Epoch 2 never starts.

I suspect the mounted Google Drive filesystem is the problem, because I had problems with Google Drive timeouts in the past. My Question is: Is there something I did wrong? Did one of the six workers die? Is there something I can do to restore the workers after the first epoch? After all, it seemed to work fine for the first epoch.

• I have the same problem appearing from time to time when fitting in multiprocessing mode. It highly depends on the queue size I use with respect to the number of workers I use. It's super annoying because I didn't identify a pattern yet of this warning (and then bug appearing) – Zaccharie Ramzi Sep 27 '19 at 15:41
• @ZaccharieRamzi One thing that seems to help in my case is to copy all training data from the mounted Drive file system to the Google Colab file system. I will add that as a solution if I don't encounter the problem anymore in the future. – theblackips Oct 3 '19 at 11:04
• I am not using Colab. However my data is indeed on a hard drive not on my system. I have too much of it however, so it's not possible for me to copy it on the system. One quick-fix that works for me is to have a low max_queue_size. – Zaccharie Ramzi Oct 3 '19 at 11:42
• Make sure the drive which contains your data is stable. Mounting google drive is not stable. – mhndlsz Oct 15 '19 at 17:56