2
$\begingroup$

I'm developing a pipeline to fit parameters for a gradient boosting classifier while also fitting the optimum number of features in a PCA model. This is the current setup:

pipe = Pipeline([
    ('reduce_dim', PCA()),
    ('classify', GradientBoostingClassifier())
])
score = {'f1': 'f1', 'accuracy': 'accuracy'}
N_FEATURES_OPTIONS = [12,13,14,15]
max_dep = [3,4,5,6]
n_est = [50,80,100, 120, 150]
min_samp = [4,5,6,10]
param_grid = [
    {
        'reduce_dim': [PCA()],
        'reduce_dim__n_components': N_FEATURES_OPTIONS,
        'classify__n_estimators': n_est,
        'classify__max_depth': max_dep,
        'classify__min_samples_split':min_samp
    }]
reducer_labels = ['PCA']

grid_adc = GridSearchCV(pipe, cv=5, n_jobs=-1, param_grid=param_grid, scoring=score, refit='accuracy')
grid_adc.fit(X_train, y_train)
grid_adc.best_params_

Which outputs:

{'classify__max_depth': 3,
 'classify__min_samples_split': 4,
 'classify__n_estimators': 50,
 'reduce_dim': PCA(copy=True, iterated_power='auto', n_components=12, random_state=None,
   svd_solver='auto', tol=0.0, whiten=False),
 'reduce_dim__n_components': 12}

Now I'd like to validate and score the model with cross validation. If I run the following:

cross_val_score(grid, X_train, y_train, cv=5, n_jobs=-1)

Does the PCA I fit from the pipeline carry over to the cross_val_score function? If so, does the cross_val_score function transform the data with the PCA every time it generates a new train/test split?

Or do I need to create a new PCA after the pipeline to fit into the cross_val_score function?

$\endgroup$
  • $\begingroup$ In cross_val_score(grid, X_train, y_train, cv=5, n_jobs=-1) should you change grid for grid_adc? $\endgroup$ – David Masip May 31 '18 at 10:01
  • $\begingroup$ Correct, that's a typo. I was cobbling together a few versions of this for the question. $\endgroup$ – Karl May 31 '18 at 15:19
2
$\begingroup$

When doing GridSearchCv, the best model is already scored. You can access it with the attribute best_score_ and get the model with best_estimator_. You do not need to re-score it in a cross validation.

Also, yes, the pipeline is entirely fitted when doing each split during the cv.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.