1
$\begingroup$

I am using Orange Data mining 3.8 to classify a dataset using leave one out cross validation.

I know that the results ($AUC$, $CA$, $F1$) are averaged; How can I get the results ($AUC$, $CA$, $F1$) of every fold?

$\endgroup$
1
$\begingroup$

For AUC, you can not. Leave-one-out uses only one data instance in the training set. Computation of AUC requires more data.

For general cross-validation (say, 10-fold cross-validation), there are several ways to examine what happened in each fold. Test & Score widget that performs cross-validation outputs a data table with predictions for each of the data instances in the test set. Together with predicted probabilities, predicted class, original class, the data also include information about the fold. You can then select the results by the fold and analyze it further, say, in a box plot.

test and score with fold selection

To compute evaluation statistics for each of the folds, use Data Sampler, chose "Cross Validation" for sampling, select the desired fold, and wire the connection between Data Sampler and Test & Score widget as shown below. Notice that in this setting the Data Sampler outputs the fold data as a sample (to be used as test data), and out-of-fold data as remaining data (to be used as training data).

cross-valiadation-based data sampling

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.