1
$\begingroup$

I have noticed integrating feature selection in a pipeline alters results.

Pipeline 1 gives slightly different results with pipeline 2. Why should this be so?

Pipeline 2

 # Integrate selection in pipeline
    select = SelectPercentile(score_func=chi2, percentile=50)
    pipeline_selection = make_pipeline(
        FunctionTransformer(preprocess_data),
        encode_transformer,
        
        poly_transformer,
        
        select,
        LogisticRegression(max_iter=10000)
        
        
       
    
    )
    # Define the parameter grid for grid search
    param_grid = {
        'logisticregression__C': [0.001, 0.01, 0.1, 1, 10, 100],
        'logisticregression__penalty': ['l1','l2'],
        'logisticregression__solver': ['liblinear', 'saga','newton-cg']
    }
    
    # Create the logistic regression grid search object with cross-validation
    grid_search = GridSearchCV(pipeline_selection, param_grid, cv=5,return_train_score=True)
    
    grid_search.fit(X_train, y_train)
    
    # Get the best parameters and best score
    best_params = grid_search.best_params_
    best_score = grid_search.best_score_
    
    print("Best Parameters:", best_params)
    print("Best Score:", best_score)

Pipeline 1

encode_transformer = make_column_transformer(
     
    (OneHotEncoder(), ['Sex','Pclass','Title']),
    
     (MinMaxScaler(), ['Age']),
    
    (FunctionTransformer(boxcox_transform), ['Fare']), 
)


# Create the polynomial transformer
poly_transformer = PolynomialFeatures(degree=2, include_bias=False)


# Create the pipeline by chaining the column transformers
pipeline = make_pipeline(
    FunctionTransformer(preprocess_data),
    encode_transformer,
    
    poly_transformer
    
    
)
poly_transformed_df = pipeline.fit_transform(X_train)


select = SelectPercentile(score_func=chi2, percentile=50)
best = select.fit(poly_transformed_df, y_train)
X_train_poly_selected = select.transform(poly_transformed_df)

parameters = {
    'C': [0.001, 0.01, 0.1, 1, 10, 100],
    'penalty': ['l2'],
    'solver': ['liblinear', 'saga','newton-cg']
}

clf = GridSearchCV(LogisticRegression(max_iter=1000), parameters, cv=5, return_train_score=False)
clf.fit(X_train_poly_selected, y_train)

print(clf.best_params_)

print(clf.best_score_)
$\endgroup$

1 Answer 1

1
$\begingroup$

Pipeline 2

If you do preprocessing (selction, transformation, ...) inside the Pipeline, it will be part of the cross-validation and only trained on the training dataset. Doing so will prevent some information leakage from the validation sets into the cross-validation models.

Schema

enter image description here This schema shows how for each model inside the cross-validation (gray box) a different feature selection is performed. In this schema, I assumed that the number of features is not fixed and, hence, one model gets 5 features.

Pipeline 1

If you do the preprocessing outside of the Pipeline, it is just performed once. This might result in different selected features as in Pipeline 2.

Schema

enter image description here

Note: What you call X_train and y_train (Schema: white table outside the gray cross-validation box) will be further split by GridSearchCV into folds and from these folds train (Schema: white) and validation (Schema: orange) sets will be build. So your y_train is not the training set that the pipeline sees in it's fit-methods

$\endgroup$
2
  • 1
    $\begingroup$ I think, I will add an image to visualize what I mean. It might take a while until I find time for it $\endgroup$
    – Broele
    Jul 7, 2023 at 10:23
  • $\begingroup$ Hope this makes it clearer $\endgroup$
    – Broele
    Jul 7, 2023 at 21:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.