I'm currently trying to train a Keras model on several large CSV files. I can fit one in memory, but not all combined. From my point of view, there are several ways to deal with this problem.
I could merge all of the data into a single CSV file and then read the CSV by the chunks of the data. This would simplify the process, however, I would not be able to shuffle the data each epoch.
Other approach that I can think of is to create a custom fit_generator. However I'm not sure how to actually implement this. Should I create a special generator for each dataset, and then cycle them? This would allow me to shuffle each dataset every epoch and even their order.
I believe that it would be more elegant to implement just a single generator, and manage all of the files inside. However to make this work, I would need to know the number of batches prior to training. That would require me to first get number of samples in each file, which would take a while. Also, I would need to sort out the issue of some lines being incorrect after reading the file with pandas. That limits me from using some fast os-level functions to get the number of lines.
Or is it possible to actually create a fit_generator without setting the number of batches prior to training? What approach would you recommend?