How does GridSearchCV use Cross Validation to produce a Model's Score?

Question

I understand Cross Validation in practice, but I'm not sure how SciKit-Learn's GridSearchCV uses it to produce an overall score/ metric for a model. For example, if I do a multi-metric evaluation, does the Cross Validation step take the mean of every metric from the split's $k$ models?

I've tried looking this up in the documentation here, and even dug around in SciKit-Learn's _search module, but I still can't find an exact answer.

Answer 1

Simple Answer

If you read the introduction of the Wikipedia Entry for Cross Validation you will see that Cross Validation typically builds $k$ models and averages their scores. SciKit-Learn replicates this behaviour.

If you pass multiple scorers to GridSearchCV, like below:

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
parameters = {'learning_rate': [0.01, 0.1, 0.2, 0.3]}
model = HistGradientBoostingClassifier()
scorers = ['accuracy', 'f1_weighted']

clf = GridSearchCV(model, parameters, verbose=10, scoring=scorers, refit='f1_weighted')

GridSearchCV will score each $k$-fold of your candidate models with the metrics you've provided it after each fit. Then, it will take the average of the $k$-fold cross validations and store that for you for every model in its cv_results_ attribute. To see the results, you can fit the model from the code above and display them with the following code:

clf.fit(X, y)
print(clf.cv_results_)

Bear in mind, when using more than one scorer, you have to deliberately select one the scorers using GridSearchCV's refit='your-scorer-choice-here' parameter.

Complex Answer

You mentioned digging around the code base, so I thought it might be worthwhile pointing you in the right direction.

The code you're looking for can be found in the sklearn.module_selection._spread module. Inside of the the abstract class BaseSearchCV there's a class method called fit. That method defines an inner function called evaluate_candidates, which is a callback function used to run the parameter sets. Parameter sets are "candidates" in the model.

evaluate_candidates runs all the candidates in parallel, and gets an output dictionary from each model. The dictionary has the following form:

{'fit_error': None,
 'test_scores': {'accuracy': 0.9666666666666667, 'f1_weighted': 0.9665831244778613},
 'n_test_samples': 30,
 'fit_time': 0.19898295402526855,
 'score_time': 0.007509946823120117}

It returns one of these for every training round it goes through. Specifically, if you have $4$ candidate models with a $k$-fold of $5$, you'll have $20$ of these outputs. The output dictionaries are bundled in a list and returned to you.

[{'fit_error': None, 'test_scores': {'accuracy': 0.9666666666666667, 'f1_weighted': 0.9665831244778613}, 'n_test_samples': 30, 'fit_time': 0.14478611946105957, 'score_time': 0.0039865970611572266},
 {'fit_error': None, 'test_scores': {'accuracy': 0.9666666666666667, 'f1_weighted': 0.9665831244778613}, 'n_test_samples': 30, 'fit_time': 0.13864803314208984, 'score_time': 0.003986358642578125}
 ... ]

These, along with your "candidate parameters" (candidate_params), are added to internal lists called all_out and all_candidate_params, which are defined outside the inner function:

all_candidate_params.extend(candidate_params)
all_out.extend(out)

Finally, these are passed to BaseSearchCV's _format_results method, along with more_results, which in your case will be None, and number of cross validation splits n_splits:

results = self._format_results(
                    all_candidate_params, n_splits, all_out, all_more_results
                )

This groups each candidate model's output, averages their values, and produces the results format that you see in cv_results_.

Hopefully that clears everything up for you. If anyone is interested in the code snippets that relate to this process, let me know and I'll post them below.