Training Keras model with multiple CSV files

Question

I'm currently trying to train a Keras model on several large CSV files. I can fit one in memory, but not all combined. From my point of view, there are several ways to deal with this problem.

I could merge all of the data into a single CSV file and then read the CSV by the chunks of the data. This would simplify the process, however, I would not be able to shuffle the data each epoch.

Other approach that I can think of is to create a custom fit_generator. However I'm not sure how to actually implement this. Should I create a special generator for each dataset, and then cycle them? This would allow me to shuffle each dataset every epoch and even their order.

I believe that it would be more elegant to implement just a single generator, and manage all of the files inside. However to make this work, I would need to know the number of batches prior to training. That would require me to first get number of samples in each file, which would take a while. Also, I would need to sort out the issue of some lines being incorrect after reading the file with pandas. That limits me from using some fast os-level functions to get the number of lines.

Or is it possible to actually create a fit_generator without setting the number of batches prior to training? What approach would you recommend?

Answer 1

I would use tensorflow 2.0 with tf.data

import tensorflow as tf

filenames = ["filename1", "filename2", ...]

dataset = tf.data.Dataset.list_files(filenames, seed=42, shuffle=True)

# this reads 5 text files at a time, skips the first row of each file
dataset.interleave(lambda filename: tf.data.TextLineDataset(filename).skip(1), cycle_length=5, num_parallel_calls=tf.data.experimental.AUTOTUNE)

for line in dataset.take(5):
    print(line)

You can then pass in the dataset objects directly into tensorflow.keras for training.

There are some good examples here but please beware that this is NOT for tensorflow 2.0, and somethings may have changed

Answer 2

This may be a potential option for you to consider. Take a sub-sample of records from each csv file. This should be perfectly fine if your data is (at least somewhat) normally distributed. If the distribution is not normal, normalize it or standardize it.

from sklearn import preprocessing
normalized_X = preprocessing.normalize(X)


from sklearn import preprocessing
standardized_X = preprocessing.scale(X)

Then,

# To get 3 random rows 
# each time it gives 3 different rows 
# df.sample(3) or 
df.sample(n = 3) 


# Fraction of rows 
# here you get .50 % of the rows 
df.sample(frac = 0.5) 

Also, consider this.

# Split the data between the Training Data and Test Data
xTrain , xTest , yTrain , yTest = train_test_split(X , y , 
                                                  test_size = 0.30 , 
                                                  random_state = 0, 
                                          ----->  stratify = y)