Simple Answer
If you read the introduction of the Wikipedia Entry for Cross Validation you will see that Cross Validation typically builds $k$ models and averages their scores. SciKit-Learn replicates this behaviour.
If you pass multiple scorers to GridSearchCV
, like below:
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
parameters = {'learning_rate': [0.01, 0.1, 0.2, 0.3]}
model = HistGradientBoostingClassifier()
scorers = ['accuracy', 'f1_weighted']
clf = GridSearchCV(model, parameters, verbose=10, scoring=scorers, refit='f1_weighted')
GridSearchCV
will score each $k$-fold of your candidate models with the metrics you've provided it after each fit. Then, it will take the average of the $k$-fold cross validations and store that for you for every model in its cv_results_
attribute. To see the results, you can fit the model from the code above and display them with the following code:
clf.fit(X, y)
print(clf.cv_results_)
Bear in mind, when using more than one scorer, you have to deliberately select one the scorers using GridSearchCV
's refit='your-scorer-choice-here'
parameter.
Complex Answer
You mentioned digging around the code base, so I thought it might be worthwhile pointing you in the right direction.
The code you're looking for can be found in the sklearn.module_selection._spread
module. Inside of the the abstract class BaseSearchCV
there's a class method called fit
. That method defines an inner function called evaluate_candidates
, which is a callback function used to run the parameter sets. Parameter sets are "candidates" in the model.
evaluate_candidates
runs all the candidates in parallel, and gets an output dictionary from each model. The dictionary has the following form:
{'fit_error': None,
'test_scores': {'accuracy': 0.9666666666666667, 'f1_weighted': 0.9665831244778613},
'n_test_samples': 30,
'fit_time': 0.19898295402526855,
'score_time': 0.007509946823120117}
It returns one of these for every training round it goes through. Specifically, if you have $4$ candidate models with a $k$-fold of $5$, you'll have $20$ of these outputs. The output dictionaries are bundled in a list and returned to you.
[{'fit_error': None, 'test_scores': {'accuracy': 0.9666666666666667, 'f1_weighted': 0.9665831244778613}, 'n_test_samples': 30, 'fit_time': 0.14478611946105957, 'score_time': 0.0039865970611572266},
{'fit_error': None, 'test_scores': {'accuracy': 0.9666666666666667, 'f1_weighted': 0.9665831244778613}, 'n_test_samples': 30, 'fit_time': 0.13864803314208984, 'score_time': 0.003986358642578125}
... ]
These, along with your "candidate parameters" (candidate_params
), are added to internal lists called all_out
and all_candidate_params
, which are defined outside the inner function:
all_candidate_params.extend(candidate_params)
all_out.extend(out)
Finally, these are passed to BaseSearchCV
's _format_results
method, along with more_results
, which in your case will be None
, and number of cross validation splits n_splits
:
results = self._format_results(
all_candidate_params, n_splits, all_out, all_more_results
)
This groups each candidate model's output, averages their values, and produces the results format that you see in cv_results_
.
Hopefully that clears everything up for you. If anyone is interested in the code snippets that relate to this process, let me know and I'll post them below.