# Do I load all files at once or one at a time?

I currently have $$1700+$$ CSV files. Each of them is in the same format and structure, give or take a row or possibly a column at the end. Each CSV is $$\approx 3.8$$ MB.

I need to perform a transformation on each file

1. Extract one data set, perform a summation and column select, then store inside a folder.
2. Extract an array, no need for column names here, then sore inside a folder.

From an algorithmic POV, is it better to load each file, perform the actions needed and then move on to the next file?

OR

Do I load all files, perform the action on all files and then store to hard drive?

I know how to do the actual process, I am after a 20,000 feet POV of dynamical programming/optimisation.

Thanks.

• Empirical question. Assuming results are the same, try both and time them – HEITZ Mar 25 at 23:31
• That is a really fair point. I am in the middle of doing that. Its like, if I had to write "I will not throw stones" x 1E6, do I write the sentence on one line and then the next line or do I write a colum for each character, say "I"x1E6, " "x1E5,...,"s"x1E6? Is there a classical problem that this is a special case of? – Jonathan Miller Mar 25 at 23:40