I am using Orange Data mining 3.8 to classify a dataset using leave one out cross validation.
I know that the results ($AUC$, $CA$, $F1$) are averaged; How can I get the results ($AUC$, $CA$, $F1$) of every fold?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
For AUC, you can not. Leave-one-out uses only one data instance in the training set. Computation of AUC requires more data.
For general cross-validation (say, 10-fold cross-validation), there are several ways to examine what happened in each fold. Test & Score widget that performs cross-validation outputs a data table with predictions for each of the data instances in the test set. Together with predicted probabilities, predicted class, original class, the data also include information about the fold. You can then select the results by the fold and analyze it further, say, in a box plot.
To compute evaluation statistics for each of the folds, use Data Sampler, chose "Cross Validation" for sampling, select the desired fold, and wire the connection between Data Sampler and Test & Score widget as shown below. Notice that in this setting the Data Sampler outputs the fold data as a sample (to be used as test data), and out-of-fold data as remaining data (to be used as training data).