3
$\begingroup$

Am trying to develop a python script which reads a large CSV file (approx 1.2 GB historical data) in chunks and performs following steps:

  1. Take backup of the file
  2. Extract new records of previous day transactions,append to original/base CSV file and store the data in dataframe.
  3. Perform mathematical operations on the big dataframe
  4. Convert big dataframe to CSV and store in same location for nex day processing.

Similar process runs next day as well and so on...

I'm getting memory exception error while processing the step 3 above (probably step 1 & 2 would have consumed most of the memory max-3GB limited space). Even if I extend the space in the server, I foresee a problem as my input file size is going to be increase daily.

I need to dip through historical data daily for mathematical operations along with daily transactions so can't avoid storing/accessing of base CSV file which contains historical data and can't afford to buy space in cloud as well.

I have used pd.read_csv for reading records from CSV file in chunks, for mathematical operations pandas and numpy has been used.

The script runs without any issue on my local machine, however I have the memory issue while processing on the server. So, as far as the code goes, it seems to be good.

I believe that if I can change the processing in steps 1 to 4 to be more efficient, the memory exception can be solved and with limited space my script can be executed.

Can someone suggest the best way to handle the above steps?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I had also faced similar issues when my python script was processing my 12GB and 35GB files. It used take weeks for processing and many times failed.

You need to check your mathematical operation, if it can work on subset of data. In my case, I finally was able to split the file in multiple pieces and run the mathematical operation on individual pieces. It helped improving the speed as well as better exceptional handling.

If you cant split your file, then check if your code can work in batches. Then you can read your file in chunks (in pd.read_csv you can supply chunk size) and then feed to your formula piece by piece.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Sandeep,i do not think my problem lies with mathematical operations was able to same size data in local machine in an hour time.As stated above,the main issue revolves around reading,backing up and the processing historical data which resulted memory exception.Infact,i do read data in chunks and perform mathematical operations on sub set of data (<50%) after validation check.But anyway,i will look into the process and see if it can be further split.If you can suggest on those lines(step1 to 4) that would be very helpfull. $\endgroup$
    – Optimizor
    Commented Sep 18, 2019 at 7:16
  • $\begingroup$ I wanted to execute my script with limited space(say 3GB) as the daily process increases size of the input file. $\endgroup$
    – Optimizor
    Commented Sep 18, 2019 at 7:17
  • $\begingroup$ the split count is not important. Important is how much data you can fit in memory. Based on this chnuksize the split will automatically calculated. Just check how much data you can fit in your RAM (keep RAM for OS aside) . $\endgroup$
    – Sandeep Bhutani
    Commented Sep 18, 2019 at 11:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.