2
$\begingroup$

When building a Pipeline I'm ending up at a scenario that can be simplified like this:

FeatureUnion(NumericalPipeline(steps), CategoricalPipeline(steps))

Since this is one intermediary step in a larger Pipeline, I'm feeding the preceding inputs into both of these and select the corresponding dtypes within the Numerical and Categorical Pipelines.

For some datasets, however, no Categorical Columns are left leading the Pipeline to fail. I've tried returning an empty list and 'None' but both of these did not result in the Pipeline skipping the "empty" CategoricalPipeline.

After further investigation it turns out that the SimpleImputer() in the CategoricalPipeline causes the error. Depending on the order of steps the following messages are shown:

ValueError: Found array with 0 feature(s) (shape=(150, 0)) while a minimum of 1 is required.

ValueError: at least one array or dtype is required

Any ideas on how to pass the Imputer when no Column is present?

$\endgroup$
3
  • $\begingroup$ ColumnTransformer seems like a better fit in this situation. But to answer the question directly, does "...leading the Pipeline to fail" mean the error gets raised in the CategoricalPipeline, or the FeatureUnion? Could you provide a MWE? $\endgroup$ Apr 27 '20 at 17:41
  • $\begingroup$ Thank you for your suggestion. While narrowing it down to produce an example i figured out that it's not the FeatureUnion but the SimpleImputer causing the error. $\endgroup$
    – Andreas
    Apr 27 '20 at 18:39
  • 1
    $\begingroup$ Adding a bit of code will help to debug your problem. $\endgroup$ Apr 28 '20 at 7:20
1
$\begingroup$

All(?) the sklearn transformers do a check on input data (check_X_y), which includes a check for an empty dataframe. You could probably monkey-patch out that check, but that seems like overkill.

Instead, ColumnTransformer seems the way to go. Its main purpose fits your situation. It deals with an empty columns selector gracefully, by just not calling fit on that transformer:

transformers_ : list
The collection of fitted transformers... In case there were no columns selected, this will be the unfitted transformer.

Unless you're removing columns earlier in the pipeline? In that case, please provide that additional context.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.