2
$\begingroup$

I'm developing a pipeline to fit parameters for a gradient boosting classifier while also fitting the optimum number of features in a PCA model. This is the current setup:

pipe = Pipeline([
    ('reduce_dim', PCA()),
    ('classify', GradientBoostingClassifier())
])
score = {'f1': 'f1', 'accuracy': 'accuracy'}
N_FEATURES_OPTIONS = [12,13,14,15]
max_dep = [3,4,5,6]
n_est = [50,80,100, 120, 150]
min_samp = [4,5,6,10]
param_grid = [
    {
        'reduce_dim': [PCA()],
        'reduce_dim__n_components': N_FEATURES_OPTIONS,
        'classify__n_estimators': n_est,
        'classify__max_depth': max_dep,
        'classify__min_samples_split':min_samp
    }]
reducer_labels = ['PCA']

grid_adc = GridSearchCV(pipe, cv=5, n_jobs=-1, param_grid=param_grid, scoring=score, refit='accuracy')
grid_adc.fit(X_train, y_train)
grid_adc.best_params_

Which outputs:

{'classify__max_depth': 3,
 'classify__min_samples_split': 4,
 'classify__n_estimators': 50,
 'reduce_dim': PCA(copy=True, iterated_power='auto', n_components=12, random_state=None,
   svd_solver='auto', tol=0.0, whiten=False),
 'reduce_dim__n_components': 12}

Now I'd like to validate and score the model with cross validation. If I run the following:

cross_val_score(grid, X_train, y_train, cv=5, n_jobs=-1)

Does the PCA I fit from the pipeline carry over to the cross_val_score function? If so, does the cross_val_score function transform the data with the PCA every time it generates a new train/test split?

Or do I need to create a new PCA after the pipeline to fit into the cross_val_score function?

$\endgroup$
2
  • $\begingroup$ In cross_val_score(grid, X_train, y_train, cv=5, n_jobs=-1) should you change grid for grid_adc? $\endgroup$ Commented May 31, 2018 at 10:01
  • $\begingroup$ Correct, that's a typo. I was cobbling together a few versions of this for the question. $\endgroup$
    – Karl
    Commented May 31, 2018 at 15:19

1 Answer 1

2
$\begingroup$

When doing GridSearchCv, the best model is already scored. You can access it with the attribute best_score_ and get the model with best_estimator_. You do not need to re-score it in a cross validation.

Also, yes, the pipeline is entirely fitted when doing each split during the cv.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.