How can I get the dataframe after scikit pipeline?

Question

I'm making many data transformations and fitting a model using scikit pipeline, but I need to extract X_train and X_test right after transformations (imputer, encoding, etc) in order to use it for another analysis. How can I get it?

This is my pipeline:

imputer_num = SimpleImputer(strategy = 'median')
imputer_cat = SimpleImputer(strategy = 'most_frequent')

XGB = XGBClassifier()
BBC = BalancedBaggingClassifier()
BRC = BalancedRandomForestClassifier()

models = [XGB, BBC, BRC]

numeric_transformer = Pipeline(steps=[
('imputer', SimpleImputer(strategy = 'median'))
,('scaler', StandardScaler())
])
    
categorical_transformer = Pipeline(steps=[
('imputer', SimpleImputer(strategy = 'most_frequent'))
,('encod', encoder)
])
    
preprocessor = ColumnTransformer(
 transformers=[
('num', numeric_transformer, numericas_all)
,('cat', categorical_transformer, categoricas_all)
])
    
for item in models:
    pipe = Pipeline(steps=[('preprocessor', preprocessor),('classifier', item)])
    model = pipe.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    test_probs = model.predict_proba(X_test)
    print(model)
    print(balanced_accuracy_score(y_test, y_pred))
    print(roc_auc_score(y_test, y_pred))

Answer 1

You can try applying your preprocessor to your X_train and X_test:

preprocessor = ColumnTransformer(
 transformers=[
('num', numeric_transformer, numericas_all)
,('cat', categorical_transformer, categoricas_all)
])

X_train_pipe = preprocessor.transform(X_train)
X_test_pipe = preprocessor.transform(X_test)

Edit:

Since you did not use any transformer that creates new columns like for example OneHotEncoder, this is very simple to get the feature names since those would be the same as the input matrix X (In case of using other encoders like the mentioned before you can use get_feature_names property)

Just to make all in a single step I'm going to add an additional step to this preprocessor:

from sklearn.preprocessing import FunctionTransformer
from sklearn.pipeline import Pipeline

names = X_train.columns.tolist()

preprocessor = ColumnTransformer(
     transformers=[
    ('num', numeric_transformer, numericas_all)
    ,('cat', categorical_transformer, categoricas_all)
    ])

pipe_preprocessor = Pipeline([("preprocessor", preprocessor), ("pandarizer", FunctionTransformer(lambda x: pd.DataFrame(x, columns = names)))]).fit(X_train)
    
X_train_pipe = pipe_preprocessor.transform(X_train)
X_test_pipe = pipe_preprocessor.transform(X_test)