What's the best way to train a model when having a very big dataset split in batches?

Question

I was wondering what is the best way to train my Keras Model, in a binary classification context.

My full dataset is composed of 2.3M rows and 120 columns, which is really big. As you would imagine, it doesn't fit into memory. To avoid memory errors, I have split this big dataset into 23 CSVs of 100.000 rows each. I have shuffled the full dataset prior to the splitting, so we have 23 i.i.d. samples.

I have two options:

• Take the first CSV, train the model with say 20 epochs (with mini-batches of 256), then take the next CSV, and train the same model with 20 more epochs, and so on and so on 23 times...
• Create a Python Generator that yields batches from the 23 CSVs in a way that at the end of the first epoch, the model has seen the whole 2,3M datapoints.

The main difference is the order in which the model sees the data. In both methods, the model will have seen 20 times the whole dataset.

• In the first method, the model sees 20 times the same 100.000 datapoints, then moves to the next step.
• In the second method, the model sees 1 time the whole dataset, then moves to the next step.

The reason I am asking this is because the first method is way easier to implement. A simple for loop can do the work. The second method requires Python Generators and mini-batch sampling from the already sampled CSVs.

Which of those way is the best to train my model ?

Answer 1

The first solution might give you a lot of headache as some convergence problem may arise from changing the dataset every n iterations. The second one seems the way to go, but before considering a more complex approach with generators, you might want to have some considerations at your memory usage.

Basically memory usage is : number of rows * number of columns * average memory usage. As it happens there are simple solutions to reduce each of those terms independently that often makes wonder as their reduction is multiplicative.

Average memory usage: this one has to be done on each datasets. Basically, working on your datatypes you can reduce the memory usage. This need to be done careffuly so as not to loose too much information. But basically reducing your float types from float64 to float32 already divide your memory usage by a factor of 2. This factor can go up to 8 as you could use small int8 or categories for some variables. (Plus it speeds up calculations, which might end up also giving you statistical performance boost). In the case of multiple csv files it is best to have a dictionnay of types that ensure general coherence.

Number of columns: with Machine Learning there is always the temptation to dump the whole database in the SotA algo and let it sort things out. This is not really a good approach. Usually it is best to start with a simple model (linear / logistic regression), including a smaller set of features. Selecting a subset of pertinent variables means that you will have to do some explanatory analysis. It will help you to handle your problem : having a better understanding of the relationship between features and target, observing problems like outliers and missing values, and being able to explain it. You can even use things like glmnet that do a bit of feature selection for you. With a dozen variables we are talking a factor 10 in memory usage reduction.

Number of rows : when building a model it is often practical to downsample it significantly, like at a 10% rate. This allows you to check you model building step by step without having to actually do all the calculations. Then you can reduce the downsampling. depending on your problem learning on 25% or 50% of your dataset might be more than enough, while giving a memory reduction of a factor 2-4.

Depending on your problem - if there is a variable that mainly drive the underlying behavior - you might want to build multiple models, one for each category of said variable (or buckets of a continuous variable). It's not always intuitive as most Data Scientist will prefer having one model, but sometimes it is more practical to use / handle / explain. And in your case it would allow for handling smaller databases.

All in all, those simple considerations would be enough to reduce your memory usage by a factor of 40-80.