The single CSV created by combining a large number of CSV files is too large to process. What options do I have?

Question

The dataset I am currently working on has more than 100 csv files, with each of size more than 250MB. These are files containing time series data captured from different locations and all the files have the same features as columns.

As I understand, I must combine these into one single csv file to use these data in a CNN, RNN or any other network and its assumed to be more than 20GB after completion. But this is an unacceptable file size to work on due to RAM and computational limitations.

1. What are the ways I can reduce the file size of csv files to import the files to a model?

2. Are there better file versions that I can convert these into, so it will not take as much space? (I tried .db version but it wasn't enough)

3. Is there a way which I haven't thought of to load the files separately to the model so that I can continue with the current versions of these csv files?

Answer 1

The specific questions 1 to 3 are actually irrelevant: the problem is not about file format, it's about the size of the data in memory. So even with some smaller file format, the data would still need to be fully encoded in memory to train the model.

Obviously the simple technical solution is to use a machine with more memory: possibly some computer server (often available in universities btw) or some cloud service.

If this is not possible, you can reduce the size of the data in different ways:

• Simply use a small subset of the data, either random or selecting only some locations and/or specific time interval. This should be sufficient for testing your code, trying to train a model, etc.
• For the time dimension, resampling the data could be an option. For example if you have data points every hour instead of every minute the data becomes 60 times smaller.

In general you should check that using all the data is really necessary for your goal. More data doesn't always improve performance. You can test this by training/testing a model based on various sizes of random subset (ablation study), and observe how performance increases with sizes (or not).