0
$\begingroup$

I'm watching a PyData talk from 2017 in which the speaker provides this example for how to use FunctionTransformer for sklearn.preprocessing

from sklearn.preprocessing import FunctionTransformer
logger = FunctionTransformer(np.log1p)
X_log = logger.transform(X)

In other words, she's applying a function over the rows of a column. I assumed this could be done more simply using .apply(). I feel that there must be something more to the reason why a data analyst would import FunctionTransformer. Could someone help me understand what differentiates the .apply() method from FunctionTransformer?

$\endgroup$
2
$\begingroup$

FunctionTransformer is useful because it allows you to apply a custom function in a pipeline. Because Pipeline() from sklearn.pipeline only works with objects that implement the .transform() and .fit() methods, you use FunctionTransformer to change your custom function to allow .transform() and/or .fit() to be used on it.

You could transform a DataFrame or Series by using .apply() (or something similar like a list comprehension), but you wouldn't be able to use that function in Pipeline() without first using Function Transformer.

(answer adapted from a DataCamp module "Multiple types of processing: FunctionTransformer" from the class "Machine Learning with the Experts: School Budgets")

Example:

# Import FunctionTransformer
from sklearn.preprocessing import FunctionTransformer

# Obtain the text data: get_text_data
get_text_data = FunctionTransformer(lambda x: x['text'], validate=False)

# Obtain the numeric data: get_numeric_data
get_numeric_data = FunctionTransformer(lambda x: x[['numeric', 'with_missing']], validate=False)

# Fit and transform the text data: just_text_data
just_text_data = get_text_data.fit_transform(sample_df)

# Fit and transform the numeric data: just_numeric_data
just_numeric_data = get_numeric_data.fit_transform(sample_df)

# Print head to check results
print('Text Data')
print(just_text_data.head())
print('\nNumeric Data')
print(just_numeric_data.head())
<script.py> output:
    Text Data
    0           
    1        foo
    2    foo bar
    3           
    4    foo bar
    Name: text, dtype: object

    Numeric Data
         numeric  with_missing
    0 -10.856306      4.433240
    1   9.973454      4.310229
    2   2.829785      2.469828
    3 -15.062947      2.852981
    4  -5.786003      1.826475

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.