1
$\begingroup$

I am using a custom transformer inside sklearn pipeline. The transformer removes lines from data set, but it seems it can only remove the lines from X, but not from y. I see the pipeline automatically calls fit_transform method of the transformer. Current implementation of sklearn 1.2 TransformerMixin method fit_transform calls fit(X, y) and then transform(X) Pipeline object in method transform also returns only X, not y. It seems to me a Transformer cannot really modify y.

What is the correct way to remove a line from both X and y sets inside Pipeline if not by Transformer?

$\endgroup$
2
  • 1
    $\begingroup$ What exactly do you mean by "lines"? It would be more helpful if you could give more details about your issue and the problem statement $\endgroup$
    – spectre
    Jul 6, 2023 at 5:13
  • $\begingroup$ @spectre remove observations from the dataset, or remove rows from the main dataframe from which both X and y are derived $\endgroup$ Jul 6, 2023 at 19:07

2 Answers 2

2
+50
$\begingroup$

(By 'removing lines', I assume you mean 'filter out samples')

In general, sklearn's transformers focus more on the features. For your case, there are 2 solutions:

  1. Filter out the samples before/after the pipeline if possible. This is an easier approach.

  2. Overwrite part of the custom transformer API. See this question as an example.

$\endgroup$
1
$\begingroup$

Scikit-learn does not support removing rows in a pipeline. One option is to use the imbalanced-learn package which is scikit-learn compatible and supports removing rows in a pipeline.

Here is an example:

from imblearn          import FunctionSampler
from imblearn.pipeline import Pipeline 
from sklearn.tree      import DecisionTreeClassifier


def filter_data(X, y):
    "Remove data from both X and y."
    return X[:-1], y[:-1] # Hard code dropping last row

pipe = Pipeline([('filter', FunctionSampler(func=filter_data)),
                 ('clf',    DecisionTreeClassifier()),])
pipe.fit(X, y)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.