1
$\begingroup$

I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below:

df.read_csv() #one instance of a df
df1 = df.drop_duplicates #another instance of a df, total 2
df1_melt = df1.melt() #another instance of a df
df1_aggregated = df1_melt.groupby()... #another instance, total 4
df_mutual = pd.merge(df1_aggregated, df1_melted) #created another instance of a df, total 5

In the example above, we created 5 dataframes and stored them in memory, but we're really only interested in one. I have read that you can reduce the reference count to these variables to 0, and then when you delete the variable, it may be garbage collected. Is there a better way to program in a manner that I avoid creating multiple copies of the data in memory?

$\endgroup$
  • $\begingroup$ It depends on what you want to do with it afterwards. Do you really need all these df later? $\endgroup$ – Catalina Chircu Mar 30 '20 at 14:26
  • $\begingroup$ Don't really need all the other dfs. However, I have heard that some reassignment eg: df = df.perform_something doesn't actually have any memory savings because the old data that is stored in memory is just unreferenced, but it is still there. I would just like to have some resources to learn how to do data wrangling in a memory efficient way $\endgroup$ – wombatkingdom Mar 30 '20 at 23:07
0
$\begingroup$

Programming best practices that are specifically usefull for ML / DS :

  • Know and handle your data types, with a dict for example, and pass this dict as and agrument to your read csv. You can get important memory usage reduction by using the right types of float / int and category instead of string. Float precision is rarely important in ML (and usually achieve better statistical performance trough faster calculations).

  • As a part of your model building process, it is usefull to do some variable selection (to reduce the number of columns) or sampling (to reduce the number of rows). Sometimes just for testing your pipeline, sometimes to get a baseline, sometimes as a long run solution.

  • You have to actively avoid multiplying you dataframes. I usually check for that in real time by seeing what memory is consumed. (Either trough command line interface like htop, or trough ad-hoc memory requests in the python environnement). This means you have to look into your functions which are duplicating your df. Then you have some work to do to find an alternative that is low memory (using vectorized calculation, using a 'in_place = True' option ... etc). During this step you also have to check that your operations are not upgrading the data types you specified in the first part.

  • Clean after yourself. You have to delete the objects you don't use (the raw data after you preprocessed them, the whole data you just splitted ... etc.). Basically using del() everywhere you can and the garbage collector gc.collect() often.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.