Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline?

Question

I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling.

I am pretty comfortable with the sklearn Pipeline object that I use for pre-processing but I am not sure if I should include data cleaning, data extraction and feature engineering steps that are typically more specific to the dataset I am working on.

My general thinking is that the pre-processing phase would include operations on the data that need to be done after splitting it so as to avoid data leakage. These would typically be:

• Scaling
• Imputing (if not by constant value)
• Encoding

On the other hand, data cleaning or feature engineering are operations that could be performed on the whole data set, for example:

def clean_price(data):
    """Clean the price feature."""
    data['Prezzo_EUR'] = data['Prezzo'].str.split('€').str[1].str.replace('.', '').astype('float')
    return data

def create_energy_class(data):
    """Create energy class feature."""
    data['Classe_energetica'] = data['Efficienza energetica'].str.extract(r'([A-G]\d?)')
    return data

These operations are very specific to the dataset and don't cause data leakage, and I don't really see any value in including them in an sklearn pipeline using the FunctionTransformer object for example. I could easily have 20+ simple operations like the ones shown above and what I do instead is build a custom pipeline connecting all of the functions:

def create_pipeline(list_functions):
    """Pipeline function for data cleaning steps."""

    def pipeline(data):
        out = data
        for function in list_functions:
            out = function(out)
        return out

    return pipeline

So I basically end up having two separate pipelines, one for operations to be performed on the whole dataset before splitting it (and that are very specific to the dataset), and one for operations after splitting (what I call pre-processing).

What is the approach to building scalable data pipelines that include data cleaning, data extraction, feature engineering, pre-processing before modeling? Are there better or more standard ways of going about this?

Also please correct me if you think my terminology is not accurate.

Answer 1

Having two separate pipelines would be more efficient for larger datasets. Version the different cleaned/standardised/preprocessed datasets and run the lean learning and prediction model on a particular cleaned dataset.

That being said, if the cleaning process is cheap, then keep it in the same pipeline. You don't want to bloat your work if it is unnecessary.

Answer 2

One of the benefits of using scikit-learn's Pipeline is that it appropriately applies the operations to both training and prediction datasets. The "whole dataset" you mention that is only the training data you currently have available. Scikit-learn's Pipeline can automatically handle additional data, such as production data.

Having two different pipelines can create unnecessary overhead and might lead to bugs in the system. It is best practice to have a single pipeline that handles all operations.