Not enough memory for operations with Pandas

Question

Wes McKinney, the author of Pandas, writes in his blog that

"... my rule of thumb for pandas is that you should have 5 to 10 times as much RAM as the size of your dataset. So if you have a 10 GB dataset, you should really have about 64, preferably 128 GB of RAM if you want to avoid memory management problems."

I frequently use Pandas with datasets not much smaller than my RAM (16GB). So I wonder, what are some practical implications of these "memory management problems"? Could anyone provide more insights into this? Does it mean it will store data in virtual memory on disk and therefore be very slow?

Answer 1

When Pandas finds it's maximum RAM limit it will freeze and kill the process, so there is no performance degradation, just a SIGKILL signal that stops the process completely. Speed of processing has more to do with the CPU and RAM speed i.e. DDR3 vs DDR4, latency, SSD vd HDD among other things.

Pandas has a strict memory limit but there are options other than just increasing RAM if you need to process large datasets.

1.- Dask

Here are certain limitations in dask.

• Dask cannot parallelize within individual tasks.
• As a distributed-computing framework, dask enables remote execution of arbitrary code. So dask workers should be hosted within trusted network only.

A Dask tutorial: https://medium.com/swlh/parallel-processing-in-python-using-dask-a9a01739902a

2.- Jax

3.- Feather Format

Language agnostic so it's usable in R and Python, can reduce the memory footprint of storage in general.

4.- Decreasing memory consumption natively in Pandas

Reducing the number of bits of memory to encode a column help specially when you use tree-based algorithms to process the data later. This is a script popularized by Kaggle.

import pandas as pd

def reduce_mem_usage(df, verbose=True):
   numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
   start_mem = df.memory_usage().sum() / 1024**2
   for col in df.columns:
       col_type = df[col].dtypes
       if col_type in numerics:
           c_min = df[col].min()
           c_max = df[col].max()
           if str(col_type)[:3] == 'int':
               if c_min > np.iinfo(np.int8).min and c_max <    np.iinfo(np.int8).max:
                   df[col] = df[col].astype(np.int8)
               elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
                   df[col] = df[col].astype(np.int16)
               elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
                   df[col] = df[col].astype(np.int32)
               elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:
                   df[col] = df[col].astype(np.int64)
           else:
               if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
                   df[col] = df[col].astype(np.float16)
               elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
                   df[col] = df[col].astype(np.float32)
               else:
                   df[col] = df[col].astype(np.float64)

   end_mem = df.memory_usage().sum() / 1024**2
   print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))
   print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))

   return df