Let us say we have a dataset with a feature such as Surname:

arr['Surname'] = ['Smith', 'Jones', 'Johnson', 'Smith']

And I would like to encode this categorical info as a new feature like:

arr['Surname_Count'] = [2, 1, 1, 2]

with the caveat that it is done within a scikit-learn pipeline. Are there quick ways to do this that do not involve rolling my own partition counting transformer?


2 Answers 2


Might be late but I found this question interesting:


import pandas as pd
from sklearn.datasets import load_iris
from sklearn.pipeline import Pipeline
from sklearn.compose import make_column_transformer, make_column_selector as selector
from sklearn.preprocessing import FunctionTransformer, MinMaxScaler

iris = load_iris()
X, _ = iris.data, iris.target
X = pd.DataFrame(X, columns= iris.feature_names)

valores = ['Smith', 'Jones', 'Johnson']

X["categoria"] = np.random.choice(a = valores, size = X.shape[0])


enter image description here

To validate results:


enter image description here

def f(series):
    mapeo = series.value_counts().to_dict()
    series = series.replace(mapeo)
    return series

preprocessor = make_column_transformer((MinMaxScaler(), selector(dtype_exclude= "object")),
                        (FunctionTransformer(lambda x: f(x)), selector(dtype_include= "object"))).fit(X)



enter image description here

Hope it helps!


You can check out Featuretools, which an open source python framework for automated feature engineering. Specifically for you, it can generate aggregation features such as count for your dataset.

After generating the new feature matrix with the desired column, you can use the matrix as you normally would within a scikit-learn pipeline.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.