memory problem due to large file

Question

I'm new in Python and definitely, I'm sure I do some mistake. Here is my problem and thanks all to your help in advance.

I have 2 files (one is Hive) and the other is csv and merging them. I have 64GB memory and I believe the csv file I create is around 25+ GB.

My problem is when I connect remotely, I see that memory usage hits to 100% and then I cannot even connect my workstation remotely and it needs a hard boot.

What I'm thinking is, when I merge these 2 tables, I like to save in csv (let's say 100,000 rows) and clean that from the memory and continues with another 100,000 rows, append to it and so on....

I'm not sure how to do this, I found some within Google search, most likely is about to read large files, but not sure after I read (merge or during merge in my case), to write every 100K chunks to a csv and clean it from the memory.

Any suggestions will help. Thanks Avi

Answer 1

I guess you are trying to use pandas, if so. Don't because you can't do that for what you want. I guess your operation needs all data to be loaded to the memory which is not possible. Try to use dask or a kind of sql. I'm not sure whether you want to do operations such as group by which needs all data to be loaded simultaneously or not. If you can do extra coding and your operations don't imply to load all data, such as finding the minimum of a column, you can use generators and specify the chunk_size of the read_csv method in pandas. But you have to do extra coding. You can also take a look at here. You can also take a look at here to figure out why sql and dask are more better for large operations.