I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling.

I am pretty comfortable with the sklearn Pipeline object that I use for pre-processing but I am not sure if I should include data cleaning, data extraction and feature engineering steps that are typically more specific to the dataset I am working on.

My general thinking is that the pre-processing phase would include operations on the data that need to be done after splitting it so as to avoid data leakage. These would typically be:

  • Scaling
  • Imputing (if not by constant value)
  • Encoding

On the other hand, data cleaning or feature engineering are operations that could be performed on the whole data set, for example:

def clean_price(data):
    """Clean the price feature."""
    data['Prezzo_EUR'] = data['Prezzo'].str.split('€').str[1].str.replace('.', '').astype('float')
    return data

def create_energy_class(data):
    """Create energy class feature."""
    data['Classe_energetica'] = data['Efficienza energetica'].str.extract(r'([A-G]\d?)')
    return data

These operations are very specific to the dataset and don't cause data leakage, and I don't really see any value in including them in an sklearn pipeline using the FunctionTransformer object for example. I could easily have 20+ simple operations like the ones shown above and what I do instead is build a custom pipeline connecting all of the functions:

def create_pipeline(list_functions):
    """Pipeline function for data cleaning steps."""

    def pipeline(data):
        out = data
        for function in list_functions:
            out = function(out)
        return out

    return pipeline

So I basically end up having two separate pipelines, one for operations to be performed on the whole dataset before splitting it (and that are very specific to the dataset), and one for operations after splitting (what I call pre-processing).

What is the approach to building scalable data pipelines that include data cleaning, data extraction, feature engineering, pre-processing before modeling? Are there better or more standard ways of going about this?

Also please correct me if you think my terminology is not accurate.


One of the benefits of using scikit-learn's Pipeline is to appropriately apply the operations to both training and prediction datasets. The "whole dataset" you mention that is only the training data you currently have available. Scikit-learn's Pipeline can automatically handle additional data, such as production data.

Having two different pipelines can create unnecessary overhead and might lead to bugs in the system. It is best practices to have a single pipeline that handles all operations.

  • $\begingroup$ So you would make custom transformers for all transformations to be applied to the data? What do you mean scikit-learn can handle production data? When I refer to the whole dataset, I mean before splitting into training/valid/test sets. Couldn't transformations on data that do not cause data leakage be applied on the entire dataset, i.e. before splitting? $\endgroup$ – LazyEval May 1 at 13:54

Having two separate pipelines would be more efficient for larger datasets. Version the different cleaned/standardised/preprocessed datasets and run the lean learning and prediction model on a particular cleaned dataset.

That being said, if the cleaning process is cheap, then keep it in the same pipeline. You don't want to bloat your work if it is unnecessary.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.