I have been searching for the deal with large CSV file read method
Its over 100gb and need to know how deal with the chunk file processing
and make concatenation faster

    import time
    filename = "../code/csv/file.csv"
    lines_number = sum(1 for line in open(filename))
    lines_in_chunk = 100# I don't know what size is better
    counter = 0
    completed = 0
    reader = pd.read_csv(filename, chunksize=lines_in_chunk)

CPU times: user 36.3 s, sys: 30.3 s, total: 1min 6s
Wall time: 1min 7s

this won't take long but the problem is concat

df = pd.concat(reader,ignore_index=True)

this part take too long and take too much memory also
is there way to make this concat process faster and efficiently ?

  • $\begingroup$ I don't understand, why reading the file by chunks if it's going to be concatenated back into a single piece of data? $\endgroup$
    – Erwan
    Jul 25, 2019 at 11:05
  • $\begingroup$ Koalas and Vaex are the way to go for huge data unless you want to try Sparkling water from H2O. $\endgroup$
    – Syenix
    Dec 25, 2019 at 23:51

1 Answer 1


Its too big file to handle by standard way. You could do it by chunk

for chunk in reader:
    chunk['col1']=chunk['col1']**2 #and so on

Or dump yours csv file to database.

number of rows

for chunk in reader: 
num_of_rows = num*lines_of_chunk

#work around in bash and python
import subprocess
subprocess.check_output(["wc","-l", "file.csv"])
  • $\begingroup$ can you show me more to handle the chunk? $\endgroup$
    – slowmonk
    Jul 25, 2019 at 13:32
  • $\begingroup$ Chunk is dataframe. You could make everything like typical dataframe in Pandas. For example, name of columns you could get from first chunk. $\endgroup$
    – fuwiak
    Jul 25, 2019 at 13:37
  • $\begingroup$ how do you know the number of rows of the reader? $\endgroup$
    – slowmonk
    Jul 25, 2019 at 16:14
  • $\begingroup$ not the chunk size the whole number of rows of original data $\endgroup$
    – slowmonk
    Jul 25, 2019 at 16:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.