We are implementing an
sklearn pipeline as follows (pseudo code):
feature_num = numeric_columns feature_cat = categorical_columns num_pipeline = Pipeline([('feature_select', DataFrameSelector(feature_num)) ,('feature_num_prep', custom_num_prep) ]) cat_pipeline = Pipeline([('feature_select', DataFrameSelector(feature_cat)) ,('feature_cat_prep', custom_cat_prep) ]) feature_pipeline = FeatureUnion(transformer_list=[ ('num_pipeline', num_pipeline) ,('cat_pipeline', cat_pipeline) ])
Since we have in the
custom_num_prep transformer the possibility to remove outliers - when we get to the feature_pipeline there is an error:
ValueError: [blocks[0,:] has incompatible row dimensions....
The reason is obvious since the
cat_pipeline has observations that the
num_pipeline has removed.
The question is:
1. is there a pythonic way to sync the two dataframes within the pipelines?
2. if not what strategy is there for such situations?