# sklearn ColumnTransformer creates new columns in output when there are overlapping columns between steps

I need to process some dataframe columns in different steps using ColumnTransformer. The first step process the date columns (timestamp) imputing missing values and the second step applies scaling to all the numeric columns (including the dates columns). In output I get a number of columns which is the sum of the numeric columns and the dates columns, but the dates columns are a subset of the numeric columns so this is not correct.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

dates_columns = ['ts_1', 'ts_2']
numeric_columns = ['ts_1', 'ts_2', 'n_1', 'n_2']

column_transformer = ColumnTransformer([
('imputer_dates', SimpleImputer(strategy='median', missing_values=0), date_columns),
('scaler', StandardScaler(), numeric_columns)
])

X_transformed = column_transformer.fit_transform(X)
print(X_transformed.shape) # Got 6 columns, but it should be 4


How to fix this?

The description says-

".................the features generated by each transformer will be concatenated to form a single feature space"

Based on this I would not expect it to "reduce" the number of columns.

On top of my mind, another pipeline which computes on dates column and feeds its output to numeric column transformation in columnTransformation.

• Hi, are there alernatives to apply a series of transformations in place to the existing columns?
– revy
May 19 '20 at 9:51
• I just updated the answer. you can make a pipeline which scale dates and then feed its output to another transformation so that it applies the transformation on scaled data. Although why are you scaling dates? here is a sample you can get some idea from-stackoverflow.com/questions/54160370/… May 19 '20 at 9:55
• Thanks, I'll give it a try! My dates are in timestamp (epoch) format, I've read somewhere that ML models work better if all the numeric columns are scaled
– revy
May 19 '20 at 10:29