How to train a model using a daunting huge training dataset

Question

I have an extremely huge dataset and I'm wondering me how could be the right way to set an experiment to use this data to train a model.

I understand that I can use data-reduction to, for instance, drop out some variables. In despite data-reduction can actually reduces the data amount, as I can see this technique is intended to improve the model training effectiveness, not to deal with the practical issues that comes out from the data amount.

One of my ideas is to suffle the whole data first and then split the data into 'small' chunks. Once I have, let's say' $N$ chuncks, I can train the same model using each chunck as follows:

initialize(M);
for(n in N) {
  D = load_chunck(n);
  M = train(M, D);
}

Although this approach can be effective to fit the experiment to computational resources at hand, I'm afraid that training the model this way can affect the model's quality by including bias from the latter chuncks. In addition, N is now a new hyperparameter to be set.

Another alternative I can see is by using statistical sampling:

D = retrieve_sampling(sampling_size);
if (D is good)
   M = train(D);

I'm wondering me if there are other ways to do it then the ones I've cited here.

Answer 1

Quite often with massive datasets the model doesn't actually need the whole data. So I think the first step is to check whether using the whole data is useful: run an ablation study where you use say 1%, then 2%, 3%, .., up to say 10% of the data (adapt the levels to your case of course). Each run consists in training on the x % subset and evaluating on a validation set (be sure to separate your real final test set before anything, this study should only use a validation set).

The goal is to estimate how much gain in performance is obtained by adding training data. Plotting the performance as a function of the amount of data should give a decent idea of the trend, even if it doesn't reach the point of maximal performance, i.e. when more data doesn't improve performance anymore. With this information you can make a better decision about how to proceed with the real training.