When Pandas finds it's maximum RAM limit it will freeze and kill the process, so there is no performance degradation, just a SIGKILL signal that stops the process completely. Speed of processing has more to do with the CPU and RAM speed i.e. DDR3 vs DDR4, latency, SSD vd HDD among other things.
Pandas has a strict memory limit but there are options other than just increasing RAM if you need to process large datasets.
1.- Dask
Here are certain limitations in dask.
- Dask cannot parallelize within individual tasks.
- As a distributed-computing framework, dask enables remote execution of arbitrary code. So dask workers should be hosted within trusted network only.
A Dask tutorial:
https://medium.com/swlh/parallel-processing-in-python-using-dask-a9a01739902a
2.- Jax
3.- Feather Format
Language agnostic so it's usable in R and Python, can reduce the memory footprint of storage in general.
4.- Decreasing memory consumption natively in Pandas
Reducing the number of bits of memory to encode a column help specially when you use tree-based algorithms to process the data later. This is a script popularized by Kaggle.
import pandas as pd
def reduce_mem_usage(df, verbose=True):
numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
start_mem = df.memory_usage().sum() / 1024**2
for col in df.columns:
col_type = df[col].dtypes
if col_type in numerics:
c_min = df[col].min()
c_max = df[col].max()
if str(col_type)[:3] == 'int':
if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:
df[col] = df[col].astype(np.int8)
elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
df[col] = df[col].astype(np.int16)
elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
df[col] = df[col].astype(np.int32)
elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:
df[col] = df[col].astype(np.int64)
else:
if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
df[col] = df[col].astype(np.float16)
elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
df[col] = df[col].astype(np.float32)
else:
df[col] = df[col].astype(np.float64)
end_mem = df.memory_usage().sum() / 1024**2
print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))
print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))
return df