1
$\begingroup$

I'm making many data transformations and fitting a model using scikit pipeline, but I need to extract X_train and X_test right after transformations (imputer, encoding, etc) in order to use it for another analysis. How can I get it?

This is my pipeline:

imputer_num = SimpleImputer(strategy = 'median')
imputer_cat = SimpleImputer(strategy = 'most_frequent')

XGB = XGBClassifier()
BBC = BalancedBaggingClassifier()
BRC = BalancedRandomForestClassifier()

models = [XGB, BBC, BRC]

numeric_transformer = Pipeline(steps=[
('imputer', SimpleImputer(strategy = 'median'))
,('scaler', StandardScaler())
])
    
categorical_transformer = Pipeline(steps=[
('imputer', SimpleImputer(strategy = 'most_frequent'))
,('encod', encoder)
])
    
preprocessor = ColumnTransformer(
 transformers=[
('num', numeric_transformer, numericas_all)
,('cat', categorical_transformer, categoricas_all)
])
    
for item in models:
    pipe = Pipeline(steps=[('preprocessor', preprocessor),('classifier', item)])
    model = pipe.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    test_probs = model.predict_proba(X_test)
    print(model)
    print(balanced_accuracy_score(y_test, y_pred))
    print(roc_auc_score(y_test, y_pred))
$\endgroup$
1
$\begingroup$

You can try applying your preprocessor to your X_train and X_test:

preprocessor = ColumnTransformer(
 transformers=[
('num', numeric_transformer, numericas_all)
,('cat', categorical_transformer, categoricas_all)
])

X_train_pipe = preprocessor.transform(X_train)
X_test_pipe = preprocessor.transform(X_test)

Edit:

Since you did not use any transformer that creates new columns like for example OneHotEncoder, this is very simple to get the feature names since those would be the same as the input matrix X (In case of using other encoders like the mentioned before you can use get_feature_names property)

Just to make all in a single step I'm going to add an additional step to this preprocessor:

from sklearn.preprocessing import FunctionTransformer
from sklearn.pipeline import Pipeline

names = X_train.columns.tolist()

preprocessor = ColumnTransformer(
     transformers=[
    ('num', numeric_transformer, numericas_all)
    ,('cat', categorical_transformer, categoricas_all)
    ])

pipe_preprocessor = Pipeline([("preprocessor", preprocessor), ("pandarizer", FunctionTransformer(lambda x: pd.DataFrame(x, columns = names)))]).fit(X_train)
    
X_train_pipe = pipe_preprocessor.transform(X_train)
X_test_pipe = pipe_preprocessor.transform(X_test)
$\endgroup$
1
  • $\begingroup$ It creates an array. Can I get cols name? $\endgroup$
    – rebar
    Oct 12 '21 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.