I have two CSV files(each of the file size is in GBs). I am trying to merge the two CSV files but every time I try to it my computer hangs. Is there no way to merge the files in chunks in pandas itself?

  • $\begingroup$ By merge, do you mean performing JOIN operations or appending one file to another? $\endgroup$ – Rohan Jul 29 '16 at 16:51
  • $\begingroup$ JOIN operation . Appending isn't that costly. $\endgroup$ – enterML Jul 29 '16 at 17:21
  • 2
    $\begingroup$ Can you hold at least one of them in RAM? If so, you can use iterate over the second frame in chunks to do your join, and append the results to a file in a loop. $\endgroup$ – user666 Jul 29 '16 at 18:08
  • $\begingroup$ AFAIK it is not possible in Python. You could use Spark with Hive. You can load data and run SQL like queries on it. $\endgroup$ – Rohan Jul 29 '16 at 18:20

No, there is not. You will have to use an alternative tool like dask, drill, spark, or a good old fashioned relational database.

| improve this answer | |

When faced with such situations (loading & appending multi-GB csv files), I found @user666's option of loading one data set (e.g. DataSet1) as a Pandas DF and appending the other (e.g. DataSet2) in chunks to the existing DF to be quite feasible.

Here is the code I implement:

import pandas as pd

amgPd = pd.DataFrame()
for chunk in pd.read_csv(path1+'DataSet1.csv', chunksize = 100000, low_memory=False):
    amgPd = pd.concat([amgPd,chunk])
| improve this answer | |
  • $\begingroup$ But pandas holds its DataFrames in memory, would you really have enough RAM for large data sets? $\endgroup$ – NoName Feb 2 at 6:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.