I have a data set with categorical features represented as string values and I want to fill-in missing values in it. I’ve tried to use sklearn’s
SimpleImputer but it takes too much time to fulfill the task as compared to pandas. Both methods produce the same output.
Here is the code to reproduce the behavior on a synthetic data:
import numpy as np import pandas as pd from sklearn.impute import SimpleImputer lst = np.array(['a', 'b', np.nan], dtype='object') arr = np.random.choice(lst, size=(10**6,1), p=[0.6, 0.3, 0.1]) ser = pd.Series(arr.ravel())
%%time imp = SimpleImputer(missing_values=np.nan, strategy='most_frequent') imp.fit_transform(arr)
Wall time: 13 s
Wall time: 64.8 ms
Things get even worse when string values are longer (e.g. ‘abc’ instead of one letter ‘a’). For numerical data pandas still outperform sklearn, but the difference is not that huge.
What am I doing wrong?