I am aware that there exists a function in keras.preprocessing.image.ImageDataGenerator called flow_from_dataframe. But this function assumes that we have one dataframe containing all the paths to the images and labels associated with it. But what if we have multiple dataframes and we want to use these dataframes on a per-epoch basis so that in each epoch we can load a different dataframe and the labels and images associated with it? Thanks in advance.

  • 1
    $\begingroup$ You can write your own generator. $\endgroup$
    – Valentas
    Commented Jun 25, 2020 at 7:07
  • $\begingroup$ Yeah i did that, but i would prefer a builtin function provided by library or framework as it reduces the scope of mistakes. $\endgroup$ Commented Jun 25, 2020 at 9:16

1 Answer 1


I believe the training will continue if you don't recompile and we put dataset only in the last step.
So, something similar to this snippet should work

train_datagen = ImageDataGenerator(rescale = 1./255, shear_range = 0.2, horizontal_flip = True)

training_set_1 = train_datagen.flow_from_dataframe(dataframe=df, color_mode='grayscale',target_size = (28, 28,1), batch_size = 2)
training_set_2 = train_datagen.flow_from_dataframe(dataframe=df, color_mode='grayscale',target_size = (28, 28,1), batch_size = 2)

model = keras.models.Sequential([ ... ])

optimizer = keras.optimizers.SGD(lr=0.2, momentum=0.9, decay=0.01)
model.compile(loss="binary_crossentropy", optimizer=optimizer,

for i in list(range(10)):
    history = model.fit(training_set_1,epochs=1)
    history = model.fit(training_set_2,epochs=1)

Definitely, learning can be very noisy based on the difference in variance in datagens.
Also,history will be reset everytime.

  • $\begingroup$ Cool, yeah the training is never forfited until we recompile the model, in which case we get a new model name. This should work. Thanks a lot. $\endgroup$ Commented Jun 25, 2020 at 9:14
  • $\begingroup$ But I guess the callbacks like EarlyStop might not work in this scenario....Let me know if there is a way to make EarlyStop in this case $\endgroup$ Commented Jun 25, 2020 at 10:45
  • $\begingroup$ You can check the loss(can get loss from history)/patience counter and break the loop after a certain count. A better option - Create custom callback - stackoverflow.com/a/55892107/2187625 and stackoverflow.com/questions/53915771/… $\endgroup$
    – 10xAI
    Commented Jun 25, 2020 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.