For large datasets in terms of rows, usually it is handled by splitting data into pieces and feeding them into the model one at a time using tf.datasets or custom generator.

However what if number of columns increases to 1000 ~ 2000? Should I just cut row into smaller pieces or is there a more efficient way?

[EDIT] assuming dimensionality reduction and other data preprocessing steps has already been done

  • $\begingroup$ Use Principal component analysis (PCA) to reduce the features before passing inputs to your model. $\endgroup$
    – DataFramed
    Oct 4, 2022 at 8:32
  • $\begingroup$ Does the data fit on disk, and if so, can it be memory mapped? (e.g. is it numerical data) Do you have a cluster of machines to train on, or just one? $\endgroup$
    – towr
    Oct 4, 2022 at 19:34
  • $\begingroup$ You could use some engineer tricks. First one is the simplest: using special structures for data loading therefore loading it in RAM by batches. You could reduce batch size if this operation doesn't affect your quality much. In Pytorch one could use DataLoader. And the second one: gradient checkpoints. It's some tradeoff of computations and memory. In torch you can find it on torch.utils.checkpoint $\endgroup$
    – taciturno
    Oct 6, 2022 at 7:04

3 Answers 3


It depends on the objective and the quality you want to reach.

2000 columns are a lot for most models and maybe all of them are not necessary or have very similar behavior.

That's why you could start with a correlation map. In many business cases, a lot of columns are not correlated at all or have null/zero values, or noise. They could be removed easily. Then the most (anti-)correlated ones could be merged together.

Finally, you could do separate study groups with specific columns to deal with different objectives.

In some cases, you can apply a column transformer to prepare the data easily.

As I don't know the data content, I can only give general pieces of advice.

  • 1
    $\begingroup$ Didn't know about column transformer, thanks! I've forgot to add but dataset is already preprocessed $\endgroup$
    – haneulkim
    Oct 4, 2022 at 8:13
  • $\begingroup$ @haneulkim - that dataset is not preprocessed for your needs! What you have is the starting point. Regardless of what has happened before, you must do your own preprocessing and dimensionality reduction to make the data suitable for your use of it. $\endgroup$
    – Paul Smith
    Oct 4, 2022 at 21:04
  • $\begingroup$ @PaulSmith Thanks paul! However this is actually after all data preprocessing, including dimensionality reduction. $\endgroup$
    – haneulkim
    Oct 5, 2022 at 13:23
  • $\begingroup$ @haneulkim does it answer your question? If not, please let me know. $\endgroup$ Oct 21, 2022 at 7:40

Very, very slowly.

For most algorithms, adding dimensions has a non-linear impact on processing time, and once you can no longer fit all the data in memory, you should prepare for an exponential increase in processing time.

Selecting your algorithm before understanding your data very rarely leads to a successful outcome and while Deep Learning can be done with massively dimensional data, it is very rarely worth the effort.

The first step in any data mining process is understanding what data you have available to you. It amazes me how many people forget to do simple data correlation exercises which can often eliminate 60%-90% of the data. Algorithms that use training and testing datasets are another way to massively reduce the amount of data you need to process to determine if there is anything there worth chasing.

Then apply some simple categorisation algorithms to determine if any interesting patterns exist. If they do, then you can use more complex (read expensive) algorithms to improve accuracy.

You could also investigate various columnar storage approaches to massively compress the memory footprint of your data set. If a column only uses seven discrete values in your data set, then it can be represented with just three bits.

In the worst case, you can apply sampling to reduce the columns and rows to something that fits in memory. If no meaningful and useful patterns are found after an arbitrary number of samples are processed, walk away from the problem saying there is nothing that can be found with the resources available.


Small Bach size. But this will take a very long time, based on the dataset size.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.