I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling.
I am pretty comfortable with the sklearn
Pipeline object that I use for pre-processing but I am not sure if I should include data cleaning, data extraction and feature engineering steps that are typically more specific to the dataset I am working on.
My general thinking is that the pre-processing phase would include operations on the data that need to be done after splitting it so as to avoid data leakage. These would typically be:
- Imputing (if not by constant value)
On the other hand, data cleaning or feature engineering are operations that could be performed on the whole data set, for example:
def clean_price(data): """Clean the price feature.""" data['Prezzo_EUR'] = data['Prezzo'].str.split('€').str.str.replace('.', '').astype('float') return data def create_energy_class(data): """Create energy class feature.""" data['Classe_energetica'] = data['Efficienza energetica'].str.extract(r'([A-G]\d?)') return data
These operations are very specific to the dataset and don't cause data leakage, and I don't really see any value in including them in an sklearn pipeline using the
FunctionTransformer object for example. I could easily have 20+ simple operations like the ones shown above and what I do instead is build a custom pipeline connecting all of the functions:
def create_pipeline(list_functions): """Pipeline function for data cleaning steps.""" def pipeline(data): out = data for function in list_functions: out = function(out) return out return pipeline
So I basically end up having two separate pipelines, one for operations to be performed on the whole dataset before splitting it (and that are very specific to the dataset), and one for operations after splitting (what I call pre-processing).
What is the approach to building scalable data pipelines that include data cleaning, data extraction, feature engineering, pre-processing before modeling? Are there better or more standard ways of going about this?
Also please correct me if you think my terminology is not accurate.