The dataset I am currently working on has more than 100 csv files, with each of size more than 250MB. These are files containing time series data captured from different locations and all the files have the same features as columns.
As I understand, I must combine these into one single csv file to use these data in a CNN, RNN or any other network and its assumed to be more than 20GB after completion. But this is an unacceptable file size to work on due to RAM and computational limitations.
What are the ways I can reduce the file size of csv files to import the files to a model?
Are there better file versions that I can convert these into, so it will not take as much space? (I tried .db version but it wasn't enough)
Is there a way which I haven't thought of to load the files separately to the model so that I can continue with the current versions of these csv files?