2
$\begingroup$

Wes McKinney, the author of Pandas, writes in his blog that

"... my rule of thumb for pandas is that you should have 5 to 10 times as much RAM as the size of your dataset. So if you have a 10 GB dataset, you should really have about 64, preferably 128 GB of RAM if you want to avoid memory management problems."

I frequently use Pandas with datasets not much smaller than my RAM (16GB). So I wonder, what are some practical implications of these "memory management problems"? Could anyone provide more insights into this? Does it mean it will store data in virtual memory on disk and therefore be very slow?

$\endgroup$

1 Answer 1

3
$\begingroup$

When Pandas finds it's maximum RAM limit it will freeze and kill the process, so there is no performance degradation, just a SIGKILL signal that stops the process completely. Speed of processing has more to do with the CPU and RAM speed i.e. DDR3 vs DDR4, latency, SSD vd HDD among other things.

Pandas has a strict memory limit but there are options other than just increasing RAM if you need to process large datasets.

1.- Dask

Here are certain limitations in dask.

  • Dask cannot parallelize within individual tasks.
  • As a distributed-computing framework, dask enables remote execution of arbitrary code. So dask workers should be hosted within trusted network only.

A Dask tutorial: https://medium.com/swlh/parallel-processing-in-python-using-dask-a9a01739902a

2.- Jax

3.- Feather Format

Language agnostic so it's usable in R and Python, can reduce the memory footprint of storage in general.

4.- Decreasing memory consumption natively in Pandas

Reducing the number of bits of memory to encode a column help specially when you use tree-based algorithms to process the data later. This is a script popularized by Kaggle.

import pandas as pd

def reduce_mem_usage(df, verbose=True):
   numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
   start_mem = df.memory_usage().sum() / 1024**2
   for col in df.columns:
       col_type = df[col].dtypes
       if col_type in numerics:
           c_min = df[col].min()
           c_max = df[col].max()
           if str(col_type)[:3] == 'int':
               if c_min > np.iinfo(np.int8).min and c_max <    np.iinfo(np.int8).max:
                   df[col] = df[col].astype(np.int8)
               elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
                   df[col] = df[col].astype(np.int16)
               elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
                   df[col] = df[col].astype(np.int32)
               elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:
                   df[col] = df[col].astype(np.int64)
           else:
               if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
                   df[col] = df[col].astype(np.float16)
               elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
                   df[col] = df[col].astype(np.float32)
               else:
                   df[col] = df[col].astype(np.float64)

   end_mem = df.memory_usage().sum() / 1024**2
   print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))
   print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))

   return df
$\endgroup$
2
  • $\begingroup$ Hi, thanks for the answer. Could you explain better what is meant by : "Dask cannot parallelize within individual tasks." ... I can't understand what this implies .. maybe a simple example would help ? $\endgroup$
    – Mike
    Jan 13, 2021 at 16:56
  • $\begingroup$ If a task is too big in data or processing time it will overwhelm individual workers, either machines or cpus. So if a single task is too demanding Dask cannot parallelize that further into further threads. There is some information here distributed.dask.org/en/latest/limitations.html $\endgroup$
    – wacax
    Feb 27, 2021 at 14:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.