I am trying to train an autoencoder CNN on Google Colab using keras. I have mounted my Google Drive which contains all training data. The training uses six workers and the data is loaded by the following custom generator class:
class ImageDataGenerator(keras.utils.Sequence):
def __init__(self, filepaths, batch_size, shape, shuffle=True):
self.shape = shape
self.batch_size = batch_size
self.filepaths = filepaths
self.shuffle = shuffle
self.on_epoch_end()
def __len__(self):
return int(np.ceil(len(self.filepaths) / self.batch_size))
def __getitem__(self, index):
from_index = index * self.batch_size
to_index = min(len(self.filepaths), (index + 1) * self.batch_size)
filepaths_temp = self.filepaths[from_index : to_index]
images = load_images(filepaths_temp) / 255.0
return images, images
def on_epoch_end(self):
if self.shuffle:
np.random.shuffle(self.filepaths)
My problem: Sometimes when I start training I get this:
Epoch 1/100
/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py:610: UserWarning: The input 1303 could not be retrieved. It could be because a worker has died.
UserWarning)
1599/1599 [==============================] - 2461s 2s/step - loss: 0.0239 - val_loss: 0.0251
Epoch 00001: val_loss improved from inf to 0.02512, saving model to /content/drive/...
Epoch 2/100
Apart from the warning epoch 1 seems to finish successfully. But after the last line is printed, nothing happens anymore. Epoch 2 never starts.
I suspect the mounted Google Drive filesystem is the problem, because I had problems with Google Drive timeouts in the past. My Question is: Is there something I did wrong? Did one of the six workers die? Is there something I can do to restore the workers after the first epoch? After all, it seemed to work fine for the first epoch.
max_queue_size
. $\endgroup$