I'm currently trying to train a Keras model on several large CSV files. I can fit one in memory, but not all combined. From my point of view, there are several ways to deal with this problem.

I could merge all of the data into a single CSV file and then read the CSV by the chunks of the data. This would simplify the process, however, I would not be able to shuffle the data each epoch.

Other approach that I can think of is to create a custom fit_generator. However I'm not sure how to actually implement this. Should I create a special generator for each dataset, and then cycle them? This would allow me to shuffle each dataset every epoch and even their order.

I believe that it would be more elegant to implement just a single generator, and manage all of the files inside. However to make this work, I would need to know the number of batches prior to training. That would require me to first get number of samples in each file, which would take a while. Also, I would need to sort out the issue of some lines being incorrect after reading the file with pandas. That limits me from using some fast os-level functions to get the number of lines.

Or is it possible to actually create a fit_generator without setting the number of batches prior to training? What approach would you recommend?


2 Answers 2


I would use tensorflow 2.0 with tf.data

import tensorflow as tf

filenames = ["filename1", "filename2", ...]

dataset = tf.data.Dataset.list_files(filenames, seed=42, shuffle=True)

# this reads 5 text files at a time, skips the first row of each file
dataset.interleave(lambda filename: tf.data.TextLineDataset(filename).skip(1), cycle_length=5, num_parallel_calls=tf.data.experimental.AUTOTUNE)

for line in dataset.take(5):

You can then pass in the dataset objects directly into tensorflow.keras for training.

There are some good examples here but please beware that this is NOT for tensorflow 2.0, and somethings may have changed

  • 1
    $\begingroup$ Thank you, I didn't know about this. Just by curiosity, why not use directly CsvDataset? Also, I believe that to preprocess the data, I can use .map() to transform the input. However, how can I work with the whole batch of data (e.g. pad it all to same length..)? $\endgroup$
    – MSKL
    Commented May 2, 2019 at 7:43
  • $\begingroup$ I added a reference link the response, but regarding batching, you should see here: cs230-stanford.github.io/… $\endgroup$
    – vgoklani
    Commented May 2, 2019 at 13:50
  • $\begingroup$ The CSV reader for tensorflow 2 is under an "experimental" section of the package, and honestly, sometimes it's just more preferable to do things directly. $\endgroup$
    – vgoklani
    Commented May 2, 2019 at 13:52

This may be a potential option for you to consider. Take a sub-sample of records from each csv file. This should be perfectly fine if your data is (at least somewhat) normally distributed. If the distribution is not normal, normalize it or standardize it.

from sklearn import preprocessing
normalized_X = preprocessing.normalize(X)

from sklearn import preprocessing
standardized_X = preprocessing.scale(X)


# To get 3 random rows 
# each time it gives 3 different rows 
# df.sample(3) or 
df.sample(n = 3) 

# Fraction of rows 
# here you get .50 % of the rows 
df.sample(frac = 0.5) 

Also, consider this.

# Split the data between the Training Data and Test Data
xTrain , xTest , yTrain , yTest = train_test_split(X , y , 
                                                  test_size = 0.30 , 
                                                  random_state = 0, 
                                          ----->  stratify = y)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.