I am trying to train an autoencoder CNN on Google Colab using keras. I have mounted my Google Drive which contains all training data. The training uses six workers and the data is loaded by the following custom generator class:
class ImageDataGenerator(keras.utils.Sequence): def __init__(self, filepaths, batch_size, shape, shuffle=True): self.shape = shape self.batch_size = batch_size self.filepaths = filepaths self.shuffle = shuffle self.on_epoch_end() def __len__(self): return int(np.ceil(len(self.filepaths) / self.batch_size)) def __getitem__(self, index): from_index = index * self.batch_size to_index = min(len(self.filepaths), (index + 1) * self.batch_size) filepaths_temp = self.filepaths[from_index : to_index] images = load_images(filepaths_temp) / 255.0 return images, images def on_epoch_end(self): if self.shuffle: np.random.shuffle(self.filepaths)
My problem: Sometimes when I start training I get this:
Epoch 1/100 /usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py:610: UserWarning: The input 1303 could not be retrieved. It could be because a worker has died. UserWarning) 1599/1599 [==============================] - 2461s 2s/step - loss: 0.0239 - val_loss: 0.0251 Epoch 00001: val_loss improved from inf to 0.02512, saving model to /content/drive/... Epoch 2/100
Apart from the warning epoch 1 seems to finish successfully. But after the last line is printed, nothing happens anymore. Epoch 2 never starts.
I suspect the mounted Google Drive filesystem is the problem, because I had problems with Google Drive timeouts in the past. My Question is: Is there something I did wrong? Did one of the six workers die? Is there something I can do to restore the workers after the first epoch? After all, it seemed to work fine for the first epoch.