# Best practice - memory management when data wrangling

I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below:

df.read_csv() #one instance of a df
df1 = df.drop_duplicates #another instance of a df, total 2
df1_melt = df1.melt() #another instance of a df
df1_aggregated = df1_melt.groupby()... #another instance, total 4
df_mutual = pd.merge(df1_aggregated, df1_melted) #created another instance of a df, total 5


In the example above, we created 5 dataframes and stored them in memory, but we're really only interested in one. I have read that you can reduce the reference count to these variables to 0, and then when you delete the variable, it may be garbage collected. Is there a better way to program in a manner that I avoid creating multiple copies of the data in memory?

• It depends on what you want to do with it afterwards. Do you really need all these df later? – Catalina Chircu Mar 30 '20 at 14:26
• Don't really need all the other dfs. However, I have heard that some reassignment eg: df = df.perform_something doesn't actually have any memory savings because the old data that is stored in memory is just unreferenced, but it is still there. I would just like to have some resources to learn how to do data wrangling in a memory efficient way – wombatkingdom Mar 30 '20 at 23:07

• You have to actively avoid multiplying you dataframes. I usually check for that in real time by seeing what memory is consumed. (Either trough command line interface like htop, or trough ad-hoc memory requests in the python environnement). This means you have to look into your functions which are duplicating your df. Then you have some work to do to find an alternative that is low memory (using vectorized calculation, using a 'in_place = True' option ... etc). During this step you also have to check that your operations are not upgrading the data types you specified in the first part.
• Clean after yourself. You have to delete the objects you don't use (the raw data after you preprocessed them, the whole data you just splitted ... etc.). Basically using del() everywhere you can and the garbage collector gc.collect() often.