I'm not really used to working with pipelines, so I'm wondering how can I use custom functions and pipelines.

Situation: I want to fill some missing values with the mean but using groups based on other feature. That's why I'm using this custom function:

def replaceNullFromGroup(From, To, variable, by):

    # 1. Create aggregation from train dataset
    From_grp = From.groupby(by)[variable].median().reset_index()

    # 2. Merge dataframes
    To_merged = To.merge(From_grp, on=by, suffixes=['_test', '_train'], how = "left")

    # 3. Create dictionaries
    to_cols = [col for col in To_merged.columns if 'test' in col]
    from_cols = [col for col in To_merged.columns if 'train' in col]
    dict_cols =dict(zip(to_cols, from_cols))

    # 4. Replace null values
    for to_col, from_col  in dict_cols.items():
        To_merged[to_col] = np.where(To_merged[to_col].isnull(), 

    # 5. Clean up dataframe    
    To_merged.drop(from_col, axis=1, inplace=True)
    To_merged.columns = To_merged.columns.str.replace('_test', '')
    return To_merged

Variables meaning:

  • From: Dataframe where I'm taking the information (Train dataset)
  • To: Dataframe where I will fill the missing values (Train and test dataset)
  • variable: variable with missing values
  • by: Variables I'm using to make groups

Can I use this function in a pipeline so I can use cross validation avoiding data leakage?

Thank you very much

  • $\begingroup$ Why dont you use sklearn default computer? It can be integrated in a pipeline and search for different techniques in CV $\endgroup$ – Carlos Mougan Apr 6 at 16:38
  • $\begingroup$ Hi @CarlosMougan . Do you mean using Sklearn imputer? scikit-learn.org/stable/modules/impute.html . I don't think it's possible to fill missing values the way I'm trying with the default methods. Am I missing something? $\endgroup$ – Daniel Apr 6 at 18:49

To include this logic into a pipeline you have to create a custom transformer. You need to ask yourself:

  1. [INIT] Are there any parameters in my logic?
    • The variable you want to impute and the category you want this imputation to be based on.
  2. [FIT] What part of the logic is related to computing what the transformation will be?
    • When you compute the median() by groups and store the data somehow for later transform.
  3. [TRANSFORM] What part of the logic transforms the data, given the parameters (in 1) and the setting that was made (in 2)?
    • When you get the parameters (access specific key in the dictionary) to retrieve what was the mean for that group, and then fill the missing value with this.

Here is an example :

from sklearn.base import BaseEstimator, TransformerMixin

class CustomImputer(BaseEstimator, TransformerMixin) : 
     def __init__(self, variable, by) : 
          #self.something enables you to include the passed parameters
          #as object attributes and use it in other methods of the class
          self.variable = variable
          self.by = by
          return self

     def fit(self, X, y=None) : 
          self.map = X.groupby(self.by)[variable].mean()
          #self.map become an attribute that is, the map of values to
          #impute in function of index (corresponding table, like a dict)
          return self

     def transform(self, X, y=None) : 
          X[variable] = X[variable].fillna(value = X[by].map(self.map))
          #Change the variable column. If the value is missing, value should 
          #be replaced by the mapping of column "by" according to the map you
          #created in fit method (self.map)
          return X

Now, it can be included in any pipeline :

#Minimal example, you could include this imputer in columns transformer to 
#apply it multiple time
pipeline = Pipeline(steps = [('myImputer', CustomImputer('variabletofill',
                              ('model', LinearRegression())])

y_pred = pipeline.fit(X_train, y_train).predict(X_test)

As you can see, the mapping is computed only based on train data. Then it is re-used to impute the missing values. It is data-leakage-proof. Here is a good article that explains how to create a custom transformer.

Hope this helps

| improve this answer | |
  • $\begingroup$ Thank you very much! Let me check how it works and the website you provided and in case I have any doubt I'll let you know. $\endgroup$ – Daniel Apr 8 at 14:40
  • $\begingroup$ Hi Again, I'm using your code but I have a problem with the function map. It seems that map only works with series, which means if you use more than one variable to create groups (by) it doesn't work. I've tried to use applymap but I can not make it work. Could you help me? $\endgroup$ – Daniel Apr 8 at 16:07
  • $\begingroup$ Hi, you should create a ColumnTransformer. My code was meant to transform one variable at the time. $\endgroup$ – Rusoiba Apr 8 at 16:45
  • $\begingroup$ Otherwise I suggest you to create a dictionary in place of just a mean value. So replace : self.mean by self.mean_dict and access this dictionary later on into the transform method. $\endgroup$ – Rusoiba Apr 8 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.