I'm reading through Hands-On Machine Learning with Scikit-Learn & TensorFlow. We're going over sklearn Transformation Pipelines for preparing data for the ML algorithms.

Here's the code (housing_num is a dataframe of the numerical attributes of the main dataframe housing):

num_pipeline = Pipeline([
    ('imputer', Imputer(strategy='median')),
    ('attribs_adder', CombinedAttributesAdder()),
    ('std_scaler', StandardScaler())

housing_num_tr = num_pipeline.fit_transform(housing_num)

The text explains, "The Pipeline constructor takes a list of name/estimator pairs defining a sequence of steps. All but the last estimator must be transformers (i.e., they must have a fit_transform() method)."

1) Why do we fit_transform() some, but only fit() without transforming others? I'm guessing this has to do with the difference between estimators and transformers, which I'm confused on in this context.

2) So when using Pipeline, you can only include one estimator that won't be transformed, and it has to be the last input? ("all but the last estimator must be transformers")




To center the data (make it have zero mean and unit standard error), you subtract the mean and then divide the result by the standard deviation.

$$z' = \frac{x-\mu}{\sigma}$$


$$z = ln(x)$$

You do that on the training set of data. But then you have to apply the same transformation to your testing set (e.g. in cross-validation), or to newly obtained examples before forecast. But you have to use the same two parameters $\mu$ and $\sigma$ (values) that you used for centering the training set.

Hence, every sklearn's transform's fit() just calculates the parameters (e.g. $\mu$ and $\sigma$ in case of StandardScaler) and saves them as an internal objects state. Afterwards, you can call its transform() method to apply the transformation to a particular set of examples.

fit_transform() joins these two steps and is used for the initial fitting of parameters on the training set $x$, but it also returns a transformed $x'$. Internally, it just calls first fit() and then transform() on the same data.

| improve this answer | |
  • $\begingroup$ Ok, that makes a lot more sense. Thank you! But I'm still confused on one statement made in the book about Pipeline: "All but the last estimator must be transformers (i.e., they must have a fit_transform() method)." Is there some rule when using sklearn Pipeline that all but the last estimator must be transformers? If so, why? Thanks. $\endgroup$ – Greg Rosen May 1 '19 at 1:12
  • $\begingroup$ In the pipeline, each intermediate step is data preparation; if running the pipeline only ran their fit methods, they wouldn't actually do anything. But the last step is often fitting a model, where transforming doesn't make any sense. $\endgroup$ – Ben Reiniger May 1 '19 at 1:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.