Am trying to develop a python script which reads a large CSV file (approx 1.2 GB historical data) in chunks and performs following steps:
- Take backup of the file
- Extract new records of previous day transactions,append to original/base CSV file and store the data in dataframe.
- Perform mathematical operations on the big dataframe
- Convert big dataframe to CSV and store in same location for nex day processing.
Similar process runs next day as well and so on...
I'm getting memory exception error while processing the step 3 above (probably step 1 & 2 would have consumed most of the memory max-3GB limited space). Even if I extend the space in the server, I foresee a problem as my input file size is going to be increase daily.
I need to dip through historical data daily for mathematical operations along with daily transactions so can't avoid storing/accessing of base CSV file which contains historical data and can't afford to buy space in cloud as well.
I have used
pd.read_csv for reading records from CSV file in chunks, for mathematical operations pandas and numpy has been used.
The script runs without any issue on my local machine, however I have the memory issue while processing on the server. So, as far as the code goes, it seems to be good.
I believe that if I can change the processing in steps 1 to 4 to be more efficient, the memory exception can be solved and with limited space my script can be executed.
Can someone suggest the best way to handle the above steps?