I need to process some dataframe columns in different steps using ColumnTransformer. The first step process the date columns (timestamp) imputing missing values and the second step applies scaling to all the numeric columns (including the dates columns). In output I get a number of columns which is the sum of the numeric columns and the dates columns, but the dates columns are a subset of the numeric columns so this is not correct.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

dates_columns = ['ts_1', 'ts_2']
numeric_columns = ['ts_1', 'ts_2', 'n_1', 'n_2']

column_transformer = ColumnTransformer([
    ('imputer_dates', SimpleImputer(strategy='median', missing_values=0), date_columns),
    ('scaler', StandardScaler(), numeric_columns)

X_transformed = column_transformer.fit_transform(X)
print(X_transformed.shape) # Got 6 columns, but it should be 4

How to fix this?


The description says-

enter image description here

".................the features generated by each transformer will be concatenated to form a single feature space"

enter image description here enter image description here

Based on this I would not expect it to "reduce" the number of columns.

On top of my mind, another pipeline which computes on dates column and feeds its output to numeric column transformation in columnTransformation.

  • $\begingroup$ Hi, are there alernatives to apply a series of transformations in place to the existing columns? $\endgroup$
    – revy
    May 19 '20 at 9:51
  • $\begingroup$ I just updated the answer. you can make a pipeline which scale dates and then feed its output to another transformation so that it applies the transformation on scaled data. Although why are you scaling dates? here is a sample you can get some idea from-stackoverflow.com/questions/54160370/… $\endgroup$ May 19 '20 at 9:55
  • $\begingroup$ Thanks, I'll give it a try! My dates are in timestamp (epoch) format, I've read somewhere that ML models work better if all the numeric columns are scaled $\endgroup$
    – revy
    May 19 '20 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.