Error while using flow_from_generator

Question

Getting this error while using flow_from_generator in keras

  63/3851 [..............................] - ETA: 6:41:59 - loss: 12.8586 - acc: 0.1930
  64/3851 [..............................] - ETA: 6:41:40 - loss: 12.8544 - acc: 0.1934Traceback (most recent call last):
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/utils/data_utils.py", line 551, in get
    inputs = self.queue.get(block=True).get()
  File "/tools/anaconda3/envs/py35/lib/python3.5/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/tools/anaconda3/envs/py35/lib/python3.5/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/utils/data_utils.py", line 391, in get_index
    return _SHARED_SEQUENCES[uid][i]
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/preprocessing/image.py", line 761, in __getitem__
    return self._get_batches_of_transformed_samples(index_array)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/preprocessing/image.py", line 1106, in _get_batches_of_transformed_samples
    interpolation=self.interpolation)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/preprocessing/image.py", line 364, in load_img
    img = img.resize(width_height_tuple, resample)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/PIL/Image.py", line 1743, in resize
    self.load()
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/PIL/ImageFile.py", line 233, in load
    "(%d bytes not processed)" % len(b))
OSError: image file is truncated (42 bytes not processed)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "convolutional.py", line 394, in <module>
    runCNNconfusion()
  File "convolutional.py", line 379, in runCNNconfusion
    epochs=epochs,verbose=1, callbacks = [MetricsCheckpoint('logs')])
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/models.py", line 1227, in fit_generator
    initial_epoch=initial_epoch)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/engine/training.py", line 2115, in fit_generator
    generator_output = next(output_generator)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/utils/data_utils.py", line 557, in get
    six.raise_from(StopIteration(e), e)
  File "<string>", line 3, in raise_from
StopIteration: image file is truncated (42 bytes not processed)

Can someone please help, totally blank on how to proceed

Answer 1

I guess based on the answer here and the point referred here your problem is not from Keras. Where you can read ... we see that PIL is reading in blocks of the file and that it expects that the blocks are going to be of a certain size. It turns out that you can ask PIL to be tolerant of files that are truncated (missing some file from the block) by changing a setting.

Somewhere before your code block, simply add the following:

from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True

...and you should be good.

EDIT: It looks like this helps for the version of PIL bundled with Pillow ("pip install pillow"), but may not work for default installations of PIL.