How to remove dataset line inside sklearn Pipeline

Question

I am using a custom transformer inside sklearn pipeline. The transformer removes lines from data set, but it seems it can only remove the lines from X, but not from y. I see the pipeline automatically calls fit_transform method of the transformer. Current implementation of sklearn 1.2 TransformerMixin method fit_transform calls fit(X, y) and then transform(X) Pipeline object in method transform also returns only X, not y. It seems to me a Transformer cannot really modify y.

What is the correct way to remove a line from both X and y sets inside Pipeline if not by Transformer?

Answer 1

(By 'removing lines', I assume you mean 'filter out samples')

In general, sklearn's transformers focus more on the features. For your case, there are 2 solutions:

1. Filter out the samples before/after the pipeline if possible. This is an easier approach.

2. Overwrite part of the custom transformer API. See this question as an example.

Answer 2

Scikit-learn does not support removing rows in a pipeline. One option is to use the imbalanced-learn package which is scikit-learn compatible and supports removing rows in a pipeline.

Here is an example:

from imblearn          import FunctionSampler
from imblearn.pipeline import Pipeline 
from sklearn.tree      import DecisionTreeClassifier


def filter_data(X, y):
    "Remove data from both X and y."
    return X[:-1], y[:-1] # Hard code dropping last row

pipe = Pipeline([('filter', FunctionSampler(func=filter_data)),
                 ('clf',    DecisionTreeClassifier()),])
pipe.fit(X, y)