I am imputing my data using simple imputer from sklearn. i want to test many different ways of applying transformations to the data. i.e for logisitcic regression i would like to

  • remove nans and replace with mode
  • replace +infs with max and -infs with min
  • use standard scaler.

then for using xgboost i would like to:

  • simply replace -infs/+infs with very large or -ve large numbers.

i have been playing with sklearn pipeline and i would like to know how i can pass the custom imputers through the pipeline? e.g:

logistic_pipeline = Pipeline( steps = [('imputer', SimpleImputer(strategy = 'most frequent') ),
                                  ( 'std_scaler', StandardScaler() ),
                        ( 'model', LinearRegression() )] )

but how do i incorprate the following function into it where i am replacing infs from the training datase (df) with the max of that column . then using this max to populate it into the test.. how can i do this using pipeline?

def replace_pos_inf(df, dftest, numeric_features):

    for col in df[numeric_features].columns:
        m = df.loc[df[col] != np.inf, col].max()

    for col in df[numeric_features].columns:
        mini = df.loc[df[col] != -np.inf, col].min()

    return df,dftest
  • $\begingroup$ Are you only trying to impute? Or is this a preprocessing step? You can explore pipelines if you don't want to transform your data in between steps. $\endgroup$ – Neels May 28 '20 at 3:24
  • $\begingroup$ no i'm trying to impute and standardize, but it seems like when i use standard scaler it converts my dataframe to array so i lose all indexs, therfore when i train how can i know that the x indices correspind to the y;s $\endgroup$ – Maths12 May 28 '20 at 8:10
  • $\begingroup$ @Neels have changed question since resolved what wa going on before $\endgroup$ – Maths12 May 28 '20 at 9:00

Since you want to save the training min/max and use those to replace inf's in the test set, you need a custom transformer. To build a robust transformer, you should use some of sklearn's validation functions. And it's best to work in numpy, since as you point out an earlier transformer in a pipeline will have already converted an input dataframe to an nparray. (You could stick with dataframes, and either convert in your transformer (losing some efficiency), or make sure your transformer always comes first in a pipeline (alright if you're the only one using your code).)

Then, here's a simple version. I'm opting not to have the numeric_features as input; you're probably better off using a ColumnTransformer to do that kind of selection for you.

from sklearn.utils import check_array
from sklearn.utils.validation import check_is_fitted
from sklearn.base import TransformerMixin, BaseEstimator

class ReplaceInf(TransformerMixin, BaseEstimator):
    '''Replace +-np.inf with the max/min finite values in each column.
    mins_ : np.ndarray
        Per-column minimum finite values.
    maxs_ : np.ndarray
        Per-column maximum finite values.

    def fit(self, X, y=None):
        # validate and convert if possible:
        X = check_array(X, force_all_finite=False)
        # using `where=np.isfinite(X)`, nan's won't affect the min/max calculation,
        #  and using `clip` to transform will preserve nan's as well.
        self.mins_ = np.amin(X, axis=0, where=np.isfinite(X), initial=np.inf)
        self.maxs_ = np.amax(X, axis=0, where=np.isfinite(X), initial=-np.inf)
        return self

    def transform(self, X):
        X = check_array(X, force_all_finite=False)
        return np.clip(X, self.mins_, self.maxs_)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.