I'm new in Python and definitely, I'm sure I do some mistake. Here is my problem and thanks all to your help in advance.
I have 2 files (one is Hive) and the other is csv and merging them. I have 64GB memory and I believe the csv file I create is around 25+ GB.
My problem is when I connect remotely, I see that memory usage hits to 100% and then I cannot even connect my workstation remotely and it needs a hard boot.
What I'm thinking is, when I merge these 2 tables, I like to save in csv (let's say 100,000 rows) and clean that from the memory and continues with another 100,000 rows, append to it and so on....
I'm not sure how to do this, I found some within Google search, most likely is about to read large files, but not sure after I read (merge or during merge in my case), to write every 100K chunks to a csv and clean it from the memory.
Any suggestions will help. Thanks Avi