I have two CSV files (each of the file size is in GBs) which I am trying to merge, but every time I do that, my computer hangs. Is there no way to merge them in chunks in pandas itself?

  • $\begingroup$ By merge, do you mean performing JOIN operations or appending one file to another? $\endgroup$
    – Rohan
    Commented Jul 29, 2016 at 16:51
  • $\begingroup$ JOIN operation . Appending isn't that costly. $\endgroup$
    – enterML
    Commented Jul 29, 2016 at 17:21
  • 2
    $\begingroup$ Can you hold at least one of them in RAM? If so, you can use iterate over the second frame in chunks to do your join, and append the results to a file in a loop. $\endgroup$
    – user666
    Commented Jul 29, 2016 at 18:08
  • $\begingroup$ AFAIK it is not possible in Python. You could use Spark with Hive. You can load data and run SQL like queries on it. $\endgroup$
    – Rohan
    Commented Jul 29, 2016 at 18:20

2 Answers 2


No, there is not. You will have to use an alternative tool like dask, drill, spark, or a good old fashioned relational database.


When faced with such situations (loading & appending multi-GB csv files), I found @user666's option of loading one data set (e.g. DataSet1) as a Pandas DF and appending the other (e.g. DataSet2) in chunks to the existing DF to be quite feasible.

Here is the code I implement:

import pandas as pd

amgPd = pd.DataFrame()
for chunk in pd.read_csv(path1+'DataSet1.csv', chunksize = 100000, low_memory=False):
    amgPd = pd.concat([amgPd,chunk])
  • $\begingroup$ But pandas holds its DataFrames in memory, would you really have enough RAM for large data sets? $\endgroup$
    – NoName
    Commented Feb 2, 2020 at 6:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.