We are implementing an sklearn pipeline
as follows (pseudo code):
feature_num = numeric_columns
feature_cat = categorical_columns
num_pipeline = Pipeline([('feature_select', DataFrameSelector(feature_num))
,('feature_num_prep', custom_num_prep)
])
cat_pipeline = Pipeline([('feature_select', DataFrameSelector(feature_cat))
,('feature_cat_prep', custom_cat_prep)
])
feature_pipeline = FeatureUnion(transformer_list=[
('num_pipeline', num_pipeline)
,('cat_pipeline', cat_pipeline)
])
Since we have in the custom_num_prep
transformer the possibility to remove outliers - when we get to the feature_pipeline there is an error:
ValueError: [blocks[0,:] has incompatible row dimensions....
The reason is obvious since the cat_pipeline
has observations that the num_pipeline
has removed.
The question is:
1. is there a pythonic way to sync the two dataframes within the pipelines?
2. if not what strategy is there for such situations?