We are implementing an sklearn pipeline as follows (pseudo code):

feature_num = numeric_columns  
feature_cat = categorical_columns  

num_pipeline = Pipeline([('feature_select', DataFrameSelector(feature_num))
                        ,('feature_num_prep', custom_num_prep)
cat_pipeline = Pipeline([('feature_select', DataFrameSelector(feature_cat))
                        ,('feature_cat_prep', custom_cat_prep)
feature_pipeline = FeatureUnion(transformer_list=[
                               ('num_pipeline', num_pipeline)
                               ,('cat_pipeline', cat_pipeline)

Since we have in the custom_num_prep transformer the possibility to remove outliers - when we get to the feature_pipeline there is an error:

ValueError: [blocks[0,:] has incompatible row dimensions....

The reason is obvious since the cat_pipeline has observations that the num_pipeline has removed.
The question is:
1. is there a pythonic way to sync the two dataframes within the pipelines?
2. if not what strategy is there for such situations?


One option is scikit-learn's ColumnTransformer applied to mixed types. ColumnTransformer is designed for the purpose of applying different preprocessing and feature extraction pipelines to different subsets of the features.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.