0
$\begingroup$

I am looking at the documentation and this guide

https://www.tensorflow.org/api_docs/python/tf/keras/utils/Sequence

https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly

And I need to write in

def __len__(self):

and

def __getitem__(self, idx):

Which gets the length of the dataset, and the getting a data sample using a particular index. So I am wondering if this is appropriate to use for multiple files, and if so, are there any best practices for this?

$\endgroup$
0
$\begingroup$

The Sequence class from keras works great with multiple files. I've personally created a custom data generator using the Sequence class to load and pre-process multiple files. You should be able to figure it out using this post, I've found it to be very easy to follow the process of creating a custom data generator step by step.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the reply. The 2nd link doesn't cover multiple files, so I still haven't figured out how to handle def __len__(self): or def __getitem__(self, idx): $\endgroup$ – SantoshGupta7 Dec 1 '19 at 20:13
  • $\begingroup$ Sorry, I thought it did but you're correct, this particular one does not. I've updated my answer with a link to a post which also explains how to create a custom keras generator using the Sequence class which does include using multiple files. $\endgroup$ – Oxbowerce Dec 1 '19 at 20:26
  • $\begingroup$ Thanks, it seems that the situation in the post is that there's multiple types of input data, and so they create a seperate generator for each one. My situation is that I have multiple files for one for one datatype, so the single generator must handle the multiple files. $\endgroup$ – SantoshGupta7 Dec 1 '19 at 21:04
  • $\begingroup$ I don't fully understand what you mean with that, but the custom generator is used for the same datatype, i.e. images. They use multiple generator instances for the training and validation dataset. Check the full generator class here. $\endgroup$ – Oxbowerce Dec 2 '19 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.