0
$\begingroup$

I have a dataset that consists of 85 feature columns and 13195 rows. Approximately 50 of these features are categorical features which I encoded using OneHotEncoder. I was reading this article about sparse data sets and was intrigued to see how changing the value of the sparse parameter when defining a OneHotEncoder object may reduce memory usage for my dataset.

Before applying OneHotEncoding to categorical features in my dataset, I have a memory usage of 9.394 MB. I found this by running this code:

    BYTES_TO_MB_DIV = 0.000001
    def print_memory_usage_of_data_frame(df):
        mem = round(df.memory_usage().sum() * BYTES_TO_MB_DIV, 3) 
        print("Memory usage is " + str(mem) + " MB")
        
    print_memory_usage_of_data_frame(dataset)

Setting OneHotEncoder spare=True:

    from sklearn.compose import ColumnTransformer
    from sklearn.preprocessing import OneHotEncoder
    from sklearn.impute import SimpleImputer
    from sklearn.pipeline import Pipeline
    
    
    numeric_transformer = Pipeline(steps=[
        ('knnImputer', KNNImputer(n_neighbors=2, weights="uniform")),
        ('scaler', StandardScaler())])
    
    categorical_transformer = Pipeline(steps=[
        ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
        ('onehot', OneHotEncoder(handle_unknown='ignore', sparse=True))])
    
    preprocessor = ColumnTransformer(transformers=[
        ('num', numeric_transformer, selector(dtype_exclude="object")),
        ('cat', categorical_transformer, selector(dtype_include="object"))
    ])
    
      
    Z = pd.DataFrame(preprocessor.fit_transform(X))
    print_memory_usage_of_data_frame(Z)

Memory usage is 25.755 MB

Then running the same code above but setting spare=False like so:

    OneHotEncoder(handle_unknown='ignore', sparse=False)

Memory usage is 25.755 MB

According to the linked article, which used the sparse option in pandas get_dummies, this should result in reduced memory storage, is this not the same for Scikit Learn's OneHotEncoder?

$\endgroup$
3
  • 1
    $\begingroup$ It may be because of the numeric portion. Try just the encoder (or just dropping the numeric pipeline from the ColumnTransformer), and try changing the sparse_threshold parameter of the ColumnTransformer. $\endgroup$ – Ben Reiniger Aug 17 '20 at 14:12
  • 2
    $\begingroup$ I don't think Pandas SparseArray format is compatible with OneHotEncoder sparse matrix format (it's scipy.sparse format I think). And I think you should not use Pandas at the end of the process, you should use it to read and wrangle csv files, but don't use it after scaling and transforming data, it will eat up a lot of memory. $\endgroup$ – CrazyElf Aug 17 '20 at 14:42
  • $\begingroup$ Thanks @BenReiniger, I removed the numeric portion and the memory usage went down to 0.106 MB. I tied changing the sparse_threshold=0.5 instead of the default 0.3 but it did not change the outcome. $\endgroup$ – sums22 Aug 18 '20 at 11:42
0
$\begingroup$

Based on @BenReiniger's comment, I removed the numeric portion from the ColumnTransformer and ran the following code:


    from sklearn.compose import ColumnTransformer
    from sklearn.preprocessing import OneHotEncoder
    from sklearn.impute import SimpleImputer
    from sklearn.pipeline import Pipeline
    
    
    categorical_transformer = Pipeline(steps=[
        ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
        ('onehot', OneHotEncoder(handle_unknown='ignore', sparse=True))])
    
    preprocessor = ColumnTransformer(transformers=[
        ('cat', categorical_transformer, selector(dtype_include="object"))
    ])
    
      
    X = pd.DataFrame(preprocessor.fit_transform(X))
    
    print_memory_usage_of_data_frame(X)

The result was Memory usage is 0.106 MB,

Running the same code above but with sparse option set to False:

    OneHotEncoder(handle_unknown='ignore', sparse=False) 

resulted in Memory usage is 20.688 MB.

So it is clear that changing the sparse parameter in OneHotEncoder does indeed reduce memory usage.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.