0
$\begingroup$

I'm running cross validation on my training data:

scoring = {'acc': 'accuracy',
           'prec_macro': 'precision_macro',
           'rec_macro': 'recall_macro',
           'f1_macro' : 'f1_macro'}
all_scores = cross_validate(clf, X_train, y_train, scoring=scoring, return_train_score=True)

This works fine but the output contains scores with keys "test_...". This suggests that the function is splitting the training data into train and test subsamples. This isn't what I want. I already have an independent test sample so I don't need to make a further split in this function.

Perhaps I'm just confused about the terms used by sklearn:

Case 1) The "test" scores are the result of running the metrics on the test set for only one cross-validation split(this is fine).

Case 2) or the "test" scores are the result of running the metrics on a completely independent test set(not what I want).

The "test" scores are returned as an array of three values in my case. This leads me to believe that the "test" scores are simply referring to the set of data that is independent from the training data for each cross validation split. And thus that test set is different for each cross validation split (this is fine and not a problem).

If the "test" scores are from a completely independent sample that is never used for training in this function, then it should only contain one value, right? But it could return three values if the metrics were run on this data for each independently trained cross validation model.

My gut says its the first case and that everything is fine. Can anyone clarify this for me?

$\endgroup$
1
$\begingroup$

Its the first case - there are 3 test_scores since the default setup is 3-fold cross validation.
Which means that the training set is split into 3 subsets and at each cross validation step the model is trained on 2 subsets and tested on the one subset left out.

Its all in the docs

$\endgroup$
  • $\begingroup$ I didn't find the documentation to be that explicit. Maybe if they said "test sets" instead of "test set" under the "scoring" parameter. But yes, case 2 doesn't make much sense anyway. Thanks for clarifying this for me. $\endgroup$ – Kevin Kraft Aug 2 '18 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.