0
$\begingroup$

I understand Cross Validation in practice, but I'm not sure how SciKit-Learn's GridSearchCV uses it to produce an overall score/ metric for a model. For example, if I do a multi-metric evaluation, does the Cross Validation step take the mean of every metric from the split's $k$ models?

I've tried looking this up in the documentation here, and even dug around in SciKit-Learn's _search module, but I still can't find an exact answer.

$\endgroup$

1 Answer 1

1
$\begingroup$

Simple Answer

If you read the introduction of the Wikipedia Entry for Cross Validation you will see that Cross Validation typically builds $k$ models and averages their scores. SciKit-Learn replicates this behaviour.

If you pass multiple scorers to GridSearchCV, like below:

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
parameters = {'learning_rate': [0.01, 0.1, 0.2, 0.3]}
model = HistGradientBoostingClassifier()
scorers = ['accuracy', 'f1_weighted']

clf = GridSearchCV(model, parameters, verbose=10, scoring=scorers, refit='f1_weighted')

GridSearchCV will score each $k$-fold of your candidate models with the metrics you've provided it after each fit. Then, it will take the average of the $k$-fold cross validations and store that for you for every model in its cv_results_ attribute. To see the results, you can fit the model from the code above and display them with the following code:

clf.fit(X, y)
print(clf.cv_results_)

Bear in mind, when using more than one scorer, you have to deliberately select one the scorers using GridSearchCV's refit='your-scorer-choice-here' parameter.

Complex Answer

You mentioned digging around the code base, so I thought it might be worthwhile pointing you in the right direction.

The code you're looking for can be found in the sklearn.module_selection._spread module. Inside of the the abstract class BaseSearchCV there's a class method called fit. That method defines an inner function called evaluate_candidates, which is a callback function used to run the parameter sets. Parameter sets are "candidates" in the model.

evaluate_candidates runs all the candidates in parallel, and gets an output dictionary from each model. The dictionary has the following form:

{'fit_error': None,
 'test_scores': {'accuracy': 0.9666666666666667, 'f1_weighted': 0.9665831244778613},
 'n_test_samples': 30,
 'fit_time': 0.19898295402526855,
 'score_time': 0.007509946823120117}

It returns one of these for every training round it goes through. Specifically, if you have $4$ candidate models with a $k$-fold of $5$, you'll have $20$ of these outputs. The output dictionaries are bundled in a list and returned to you.

[{'fit_error': None, 'test_scores': {'accuracy': 0.9666666666666667, 'f1_weighted': 0.9665831244778613}, 'n_test_samples': 30, 'fit_time': 0.14478611946105957, 'score_time': 0.0039865970611572266},
 {'fit_error': None, 'test_scores': {'accuracy': 0.9666666666666667, 'f1_weighted': 0.9665831244778613}, 'n_test_samples': 30, 'fit_time': 0.13864803314208984, 'score_time': 0.003986358642578125}
 ... ]

These, along with your "candidate parameters" (candidate_params), are added to internal lists called all_out and all_candidate_params, which are defined outside the inner function:

all_candidate_params.extend(candidate_params)
all_out.extend(out)

Finally, these are passed to BaseSearchCV's _format_results method, along with more_results, which in your case will be None, and number of cross validation splits n_splits:

results = self._format_results(
                    all_candidate_params, n_splits, all_out, all_more_results
                )

This groups each candidate model's output, averages their values, and produces the results format that you see in cv_results_.

Hopefully that clears everything up for you. If anyone is interested in the code snippets that relate to this process, let me know and I'll post them below.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.