7
$\begingroup$

I am having issues with scikit-learn converting dataframes to numpy arrays.

For instance, the following code

from sklearn.impute import SimpleImputer
import pandas as pd

df = pd.DataFrame(dict(
    x=[1, 2, np.nan],
    y=[2, np.nan, 0]
))

SimpleImputer().fit_transform(df)

Returns

array([[1. , 2. ],
       [2. , 1. ],
       [1.5, 0. ]])

Is there a way to use an imputer that returns a pandas dataframe instead of a numpy array? Is there a scikit-learn implementation for that? I am aware of sklearn-pandas, but the interface is kind of different.

$\endgroup$
1

5 Answers 5

3
$\begingroup$

Since sklearn Version 1.2, set_output can be configured per estimator by calling the set_output method or globally by setting set_config(transform_output="pandas")

See Release Highlights for scikit-learn 1.2 - Pandas output with set_output API

Example for set_output():

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().set_output(transform="pandas")

Example for set_config():

from sklearn import set_config
set_config(transform_output="pandas")
$\endgroup$
5
$\begingroup$

Might be late but for anyone with the same question the answer (as almost everything with Scikit-learn) is the usage of Pipelines

from sklearn.impute import SimpleImputer
from sklearn.preprocessing import FunctionTransformer
from sklearn.pipeline import Pipeline
import pandas as pd

df = pd.DataFrame(dict(
    x=[1, 2, np.nan],
    y=[2, np.nan, 0]
))

imputer = Pipeline([("imputer", SimpleImputer()),
                    ("pandarizer",FunctionTransformer(lambda x: pd.DataFrame(x, columns = ["x", "y"])))])

imputer.fit_transform(df)
$\endgroup$
2
  • $\begingroup$ Unfortunately, this does not answer the question, because you are using pre-defined column names ['x', 'y']. The problem is that ColumnTransformer suggests names of the output columns in transformer.get_feature_names_out(). There is no easy way to pass this along within the pipeline. (At least I don't know any). $\endgroup$ Apr 5, 2022 at 21:22
  • 1
    $\begingroup$ A workaround would be using sklearndf a library aimed to solve the problem of returning a data frame in a pipeline $\endgroup$
    – Multivac
    Apr 5, 2022 at 23:48
3
$\begingroup$

[Update: As the answer just above describes, this feature has been implemented (in this pull request)].

As of 05 April 2022, this is not available in scikit-learn.

The good news is:

Hopefully, scikit-learn will make working with pandas dataframes more convenient soon.

$\endgroup$
1
$\begingroup$

The following code snippet returns a Pandas DataFrame, but overwrites the original DataFrame values:

from sklearn.impute import SimpleImputer
imp = SimpleImputer(strategy='mean')
cols = df.columns
df[cols] = imp.fit_transform(df[cols])

Note that I'm not sure whether this consumes any additional memory.

$\endgroup$
0
$\begingroup$

What you could do, is to rewrite your favorite preprocessing functions into new custom transformers. This might take time to rewrite, but it surely is helpful when you want everything to be as a dataframe. For example consider an example of a StandardScaler:

class DFStandardScaler(TransformerMixin):
    def __init__(self):
        self.ss = None
        self.mean_ = None
        self.scale_ = None
    def fit(self, X, y=None):
        self.ss = StandardScaler()
        self.ss.fit(X)
        self.mean_ = pd.Series(self.ss.mean_, index=X.columns)
        self.scale_ = pd.Series(self.ss.scale_, index=X.columns)
        return self
    def transform(self, X) -> pd.DataFrame:
        # assumes X is a DataFrame
        Xss = self.ss.transform(X)
        Xscaled = pd.DataFrame(Xss, index=X.index, columns=X.columns)
        return Xscaled
    def __str__(self):
         return "DF_StandardScaler"
    def __repr__(self):
         return "DF_StandardScaler"

Using the following as DFStandardScaler().fit_transform(df) would return the same dataframe which was provided. The only issue is that this example would expect a df with column names, but it wouldn't be hard to set column names from scratch.

Here's the sklearn's documentation on custom transformers: https://scikit-learn.org/stable/modules/preprocessing.html#custom-transformers

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.