I'm running cross validation on my training data:
scoring = {'acc': 'accuracy',
'prec_macro': 'precision_macro',
'rec_macro': 'recall_macro',
'f1_macro' : 'f1_macro'}
all_scores = cross_validate(clf, X_train, y_train, scoring=scoring, return_train_score=True)
This works fine but the output contains scores with keys "test_...". This suggests that the function is splitting the training data into train and test subsamples. This isn't what I want. I already have an independent test sample so I don't need to make a further split in this function.
Perhaps I'm just confused about the terms used by sklearn:
Case 1) The "test" scores are the result of running the metrics on the test set for only one cross-validation split(this is fine).
Case 2) or the "test" scores are the result of running the metrics on a completely independent test set(not what I want).
The "test" scores are returned as an array of three values in my case. This leads me to believe that the "test" scores are simply referring to the set of data that is independent from the training data for each cross validation split. And thus that test set is different for each cross validation split (this is fine and not a problem).
If the "test" scores are from a completely independent sample that is never used for training in this function, then it should only contain one value, right? But it could return three values if the metrics were run on this data for each independently trained cross validation model.
My gut says its the first case and that everything is fine. Can anyone clarify this for me?
