sklearn SimpleImputer too slow for categorical data represented as string values

Question

I have a data set with categorical features represented as string values and I want to fill-in missing values in it. I’ve tried to use sklearn’s SimpleImputer but it takes too much time to fulfill the task as compared to pandas. Both methods produce the same output.

Here is the code to reproduce the behavior on a synthetic data:

import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer

lst = np.array(['a', 'b', np.nan], dtype='object')
arr = np.random.choice(lst, size=(10**6,1), p=[0.6, 0.3, 0.1])
ser = pd.Series(arr.ravel())

Using SimpleImputer:

%%time
imp = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
imp.fit_transform(arr)

Wall time: 13 s

Using pandas:

%%time
ser.fillna(value=ser.mode()[0])

Wall time: 64.8 ms

Things get even worse when string values are longer (e.g. ‘abc’ instead of one letter ‘a’). For numerical data pandas still outperform sklearn, but the difference is not that huge.

What am I doing wrong?

Answer 1

Searching the source code of Sklearn for SimpleImputer (with strategy= "most_frequent"), the most frequent value is calculated within a loop in python, therefore that is the part of code that is so slow. In the source code of SimpleImputer there is also the comment that explains why they do not use the scipy.stats.mstats.mode, which is mush faster:

scipy.stats.mstats.mode cannot be used because it will no work properly if the first element is masked and if its frequency is equal to the frequency of the most frequent valid element. See https://github.com/scipy/scipy/issues/2636

So if you want to use the SimpleImputer with this strategy, a faster way would be to use the "constant" strategy and pass the most frequent value by yourself (ser.mode()[0]) then the time is almost the same:

t0 = time.time()
imp = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
imp.fit_transform(arr)
print('Simple Imputer (Most Frequent) Time Elapsed:', time.time()-t0)

t0 = time.time()
imp = SimpleImputer(missing_values=np.nan, strategy='constant', fill_value=ser.mode()[0])
imp.fit_transform(arr)
print('Simple Imputer (Constant) Time Elapsed:', time.time()-t0)

t0 = time.time()
ser.fillna(value=ser.mode()[0])
print('Pandas Time Elapsed:', time.time()-t0)

And the time elapsed for each strategy:

Simple Imputer (Most Frequent) Time Elapsed: 14.320188045501709
Simple Imputer (Constant) Time Elapsed: 0.052472829818725586
Pandas Time Elapsed: 0.04726815223693848