I'm new in Python and definitely, I'm sure I do some mistakes. Here is my problem and thanks all for your help in advance.
I have 2 files (one is Hive) and the other is CSV
and merging them. I have 64GB memory and I believe the CSV
file I create is around 25+ GB.
My problem is when I connect remotely, I see that memory usage hits to 100%, and then I cannot even connect my workstation remotely and it needs a hard boot.
What I'm thinking is,
when I merge these 2 tables, I like to save in CSV
(let's say 100,000 rows) and clean that from the memory and continues with another 100,000 rows, append to it, and so on...
I'm not sure how to do this, I found some within Google search, most likely is about to read large files, but not sure after I read (merge or during merge in my case), to write every 100K chunks to a CSV
and clean it from the memory.
Any suggestions will help.