Best practice - memory management when data wrangling

Question

I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below:

df.read_csv() #one instance of a df
df1 = df.drop_duplicates #another instance of a df, total 2
df1_melt = df1.melt() #another instance of a df
df1_aggregated = df1_melt.groupby()... #another instance, total 4
df_mutual = pd.merge(df1_aggregated, df1_melted) #created another instance of a df, total 5

In the example above, we created 5 dataframes and stored them in memory, but we're really only interested in one. I have read that you can reduce the reference count to these variables to 0, and then when you delete the variable, it may be garbage collected. Is there a better way to program in a manner that I avoid creating multiple copies of the data in memory?

Answer 1

Programming best practices that are specifically usefull for ML / DS :

• Know and handle your data types, with a dict for example, and pass this dict as and agrument to your read csv. You can get important memory usage reduction by using the right types of float / int and category instead of string. Float precision is rarely important in ML (and usually achieve better statistical performance trough faster calculations).

• As a part of your model building process, it is usefull to do some variable selection (to reduce the number of columns) or sampling (to reduce the number of rows). Sometimes just for testing your pipeline, sometimes to get a baseline, sometimes as a long run solution.

• You have to actively avoid multiplying you dataframes. I usually check for that in real time by seeing what memory is consumed. (Either trough command line interface like htop, or trough ad-hoc memory requests in the python environnement). This means you have to look into your functions which are duplicating your df. Then you have some work to do to find an alternative that is low memory (using vectorized calculation, using a 'in_place = True' option ... etc). During this step you also have to check that your operations are not upgrading the data types you specified in the first part.

• Clean after yourself. You have to delete the objects you don't use (the raw data after you preprocessed them, the whole data you just splitted ... etc.). Basically using del() everywhere you can and the garbage collector gc.collect() often.