sklearn ColumnTransformer creates new columns in output when there are overlapping columns between steps

Question

I need to process some dataframe columns in different steps using ColumnTransformer. The first step process the date columns (timestamp) imputing missing values and the second step applies scaling to all the numeric columns (including the dates columns). In output I get a number of columns which is the sum of the numeric columns and the dates columns, but the dates columns are a subset of the numeric columns so this is not correct.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

dates_columns = ['ts_1', 'ts_2']
numeric_columns = ['ts_1', 'ts_2', 'n_1', 'n_2']

column_transformer = ColumnTransformer([
    ('imputer_dates', SimpleImputer(strategy='median', missing_values=0), date_columns),
    ('scaler', StandardScaler(), numeric_columns)
])

X_transformed = column_transformer.fit_transform(X)
print(X_transformed.shape) # Got 6 columns, but it should be 4

How to fix this?

Answer 1

The description says-

".................the features generated by each transformer will be concatenated to form a single feature space"

Based on this I would not expect it to "reduce" the number of columns.

On top of my mind, another pipeline which computes on dates column and feeds its output to numeric column transformation in columnTransformation.

Answer 2

One option is to group overlapping columns under a separate pipeline.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

X = pd.DataFrame({'ts_1': np.random.randint(0,1000,100),
                  'ts_2': np.random.randint(0,1000,100),
                  'n_1': np.random.randint(0,1000,100),
                  'n_2': np.random.randint(0,1000,100)
                })

dates_columns = ['ts_1', 'ts_2']
numeric_columns = ['n_1', 'n_2']

dates_pipeline = Pipeline(steps=[
    ('imputer_dates', SimpleImputer(strategy='median', missing_values=0)),
    ('scaler', StandardScaler())
])

column_transformer = ColumnTransformer([
    ('dates_pipeline', dates_pipeline, dates_columns),
    ('scaler', StandardScaler(), numeric_columns)
])

X_transformed = column_transformer.fit_transform(X)
print(X_transformed.shape)