How can I count the number of occurrences of a category in dataset as part of an Sklearn Pipeline

Question

Let us say we have a dataset with a feature such as Surname:

arr['Surname'] = ['Smith', 'Jones', 'Johnson', 'Smith']

And I would like to encode this categorical info as a new feature like:

arr['Surname_Count'] = [2, 1, 1, 2]

with the caveat that it is done within a scikit-learn pipeline. Are there quick ways to do this that do not involve rolling my own partition counting transformer?

Answer 1

Might be late but I found this question interesting:

Try:

import pandas as pd
from sklearn.datasets import load_iris
from sklearn.pipeline import Pipeline
from sklearn.compose import make_column_transformer, make_column_selector as selector
from sklearn.preprocessing import FunctionTransformer, MinMaxScaler

iris = load_iris()
X, _ = iris.data, iris.target
X = pd.DataFrame(X, columns= iris.feature_names)

valores = ['Smith', 'Jones', 'Johnson']

np.random.seed(42)
X["categoria"] = np.random.choice(a = valores, size = X.shape[0])

Gives:

To validate results:

X.categoria.value_counts()

def f(series):
    mapeo = series.value_counts().to_dict()
    series = series.replace(mapeo)
    return series


preprocessor = make_column_transformer((MinMaxScaler(), selector(dtype_exclude= "object")),
                        (FunctionTransformer(lambda x: f(x)), selector(dtype_include= "object"))).fit(X)

pd.DataFrame(preprocessor.transform(X))

Returns:

Hope it helps!

Answer 2

You can check out Featuretools, which an open source python framework for automated feature engineering. Specifically for you, it can generate aggregation features such as count for your dataset.

After generating the new feature matrix with the desired column, you can use the matrix as you normally would within a scikit-learn pipeline.