I am looking at the documentation and this guide



And I need to write in

def __len__(self):


def __getitem__(self, idx):

Which gets the length of the dataset, and the getting a data sample using a particular index. So I am wondering if this is appropriate to use for multiple files, and if so, are there any best practices for this?


1 Answer 1


The Sequence class from keras works great with multiple files. I've personally created a custom data generator using the Sequence class to load and pre-process multiple files. You should be able to figure it out using this post, I've found it to be very easy to follow the process of creating a custom data generator step by step.

  • $\begingroup$ Thanks for the reply. The 2nd link doesn't cover multiple files, so I still haven't figured out how to handle def __len__(self): or def __getitem__(self, idx): $\endgroup$ Dec 1, 2019 at 20:13
  • $\begingroup$ Sorry, I thought it did but you're correct, this particular one does not. I've updated my answer with a link to a post which also explains how to create a custom keras generator using the Sequence class which does include using multiple files. $\endgroup$
    – Oxbowerce
    Dec 1, 2019 at 20:26
  • $\begingroup$ Thanks, it seems that the situation in the post is that there's multiple types of input data, and so they create a seperate generator for each one. My situation is that I have multiple files for one for one datatype, so the single generator must handle the multiple files. $\endgroup$ Dec 1, 2019 at 21:04
  • $\begingroup$ I don't fully understand what you mean with that, but the custom generator is used for the same datatype, i.e. images. They use multiple generator instances for the training and validation dataset. Check the full generator class here. $\endgroup$
    – Oxbowerce
    Dec 2, 2019 at 20:52
  • $\begingroup$ This is dangerously close to a link only answer. $\endgroup$ May 18, 2021 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.