# Custom keras dataset generator not accepted by fit_generator

I've got a problem trying to use generators in Keras. I've got TensorFlow 2.1 with Python 3.7. I'm using the Keras version bundled with TensorFlow

I have defined a class for my generator derived from tf.keras.models.Sequential:

class RandomFramesFromPathsToVideos(tf.keras.models.Sequential):
def __init__(self, x_set, y_set, number_of_videos_per_batch, ort_session, frames_per_video=25, type_of_frame='cropped_frame'):
self.x, self.y = x_set, y_set
self.batch_size = number_of_videos_per_batch * frames_per_video
self.number_of_videos_per_batch = number_of_videos_per_batch
self.frames_per_video = frames_per_video
self.type_of_frame = type_of_frame
self.ort_session = ort_session

def __len__(self):
return int(np.ceil(len(self.x) / float(self.number_of_videos_per_batch)))

def __getitem__(self, idx):

bla, bla, bla...

return (np.array(devol_x, dtype=np.float32), np.array(devol_y, dtype=np.float32))


and it works since I'm perfectly able to train my model item by item using a crafted loop like this:

ds_train = RandomFramesFromPathsToVideos(x_set = set_of_paths, y_set = set_of_labels,
number_of_videos_per_batch=4,
ort_session = ort_session)

for i in range(0:len(ds_train)):
(x,y)=ds_train.__getitem__(i)
model.fit(x,y)


but when I try to use the generator as it should be used:

model.fit_generator(ds_train)


(which is now equivalent in TF2.1 to model.fit(ds_train), I know), I get the error:

ValueError: Failed to find data adapter that can handle input: <class 'functions.RandomFramesFromPathsToVideos'>, <class 'NoneType'>


I've tracked the error up to this point in internal code of TensorFlow:

def select_data_adapter(x, y):
"""Selects a data adapter than can handle a given x and y."""
# TODO(scottzhu): This should be a less implementation-specific error.
raise ValueError(
"Failed to find data adapter that can handle "
"input: {}, {}".format(
_type_name(x), _type_name(y)))
raise RuntimeError(
"Data adapters should be mutually exclusive for "
"handling inputs. Found multiple adapters {} to handle "
"input: {}, {}".format(


result = {list: 7} [<class 'tensorflow.python.keras.engine.data_adapter.ListsOfScalarsDataAdapter'>, <class 'tensorflow.python.keras.engine.data_adapter.TensorLikeDataAdapter'>, <class 'tensorflow.python.keras.engine.data_adapter.GenericArrayLikeDataAdapter'>, <class 'tensor
__len__ = {int} 7


I don't understand why my generator doesn't work. Has anyone any idea about what I'm doing wrong?. My dataset is a huge bunch of videos and is impossible to fit in memory by any means. Also, I'm doing non standard transformations like locating and cropping elements that appear in the frames so I can't use an standard generator that may be already implemented in Keras.

Thanks a lot

I was following the instructions on a tutorial but it seems that there was a mistake or that the internal code of TensorFlow has changed since it was written. I was defining my class like this:

class DataGenerator(tf.keras.utils.Sequence):


and later on I tried also this:

class DataGenerator(tf.keras.models.Sequential):


(which, by the way, is completely different and is totally wrong)

Inspecting the code for some of the adapters, I saw this:

class KerasSequenceAdapter(GeneratorDataAdapter):
"""Adapter that handles keras.utils.Sequence."""

@staticmethod
def can_handle(x, y=None):
return isinstance(x, data_utils.Sequence)


and also for my other suspect adapter:

class GeneratorDataAdapter(DataAdapter):
"""Adapter that handles python generators and iterators."""

@staticmethod
def can_handle(x, y=None):
return ((hasattr(x, "__next__") or hasattr(x, "next"))
and hasattr(x, "__iter__")
and not isinstance(x, data_utils.Sequence))


where data_utils is defined as:

from tensorflow.python.keras.utils import data_utils


So, I changed the definition of my class to this:

from tensorflow.python.keras.utils import data_utils
class RandomFramesFromPathsToVideos(data_utils.Sequence):


and now it works.

I've got, so, two options. Using a subclass from data_utils.Sequence is one of them. In that case, we are talking about a "Keras generator" (handled in TensorFlow by KerasSequenceAdapter) in which you need to define two methods: __len__ and __getitem__

The other one is building a new class NOT DERIVED from data_utils.Sequence, and defining the methods __iter__ and __next__ (or simply next). In that case we are defining an standard Python generator, which will be handled by GeneratorDataAdapter inside TensorFlow

Please, bear in mind that a Keras generator is not the same thing as a Python generator. At first, that confused me a lot.