4
$\begingroup$

Say I'm trying to classify a medical condition.

Theres only two classes: Sick and Healthy.

I build a model and I can't split the data because I don't want data from the same patient being in training and test set. So I elect to use Leave-One-Subject-Out, training the model on all subject except one and testing on the left out subject.

So for each test set I have one subject and they are either healthy or sick. So the confusion matrix only contains one class where precision is technically one every time and recall equals accuracy.

I've been reading some papers that claim to use leave-subject-out training and test splits for tasks where patients either have a medical condition or do not. I've seen papers that report accuracy, recall, and precision but I don't understand how you could have precision be less than one if each subject only contains one class. I doubt these papers are lying because I've seen this more than once.

I just want to know whats going on here for them to get precision values that are less than one. Are they doing some kind of averaging or am I missing something and thinking about this in the wrong way? None of the papers explain this either.

$\endgroup$
2
  • 1
    $\begingroup$ Why is accuracy not the best measure for assessing classification models? Every criticism raised against accuracy at that thread applies equally to sensitivity, specificity etc. $\endgroup$ – Stephan Kolassa Mar 29 at 9:35
  • $\begingroup$ @StephanKolassa Thanks for the comment. I've seen this famous post many times haha and I understand the criticisms but in this case I'm just trying to understand how the precision is being calculated in this scenario. Many papers especially in this area still uses these metrics so that's why I'm curious. $\endgroup$ – IsmailE Mar 29 at 10:47
4
$\begingroup$

Your reasoning is correct that the gold standard class is the same for all the instances in a single leave-one-out test set (under the assumption that the test patient cannot become sick at some point in time, thus having both healthy and sick status).

What you're missing is the aggregation across multiple test sets: a full leave-one-out experiment repeats this process for every single patient, i.e. if there are N patients then there are N unique pairs of (training set, test set). Here is a pseudo code to show this clearly:

correct,incorrect=0    
for every patient p:
    training_set = all the patients except p
    test set = patient p
    model = train(training_set)
    // to simplify I count the patient as one instance, it's easy to count instances instead
    if model.predict(test_set) == gold_standard(p):
       correct += 1
    else:
       incorrect += 1 
accuracy = correct / (correct + incorrect)

The calculation across all the patients can lead to some predictions being correct and some others being incorrect, this is why the accuracy, precision or recall can be any value between 0 and 1.

$\endgroup$
6
  • $\begingroup$ Thank you for the quick response. So just to check if I understand you correctly I would just keep a running count across all samples for all patients? I'm just confused because if I use the equation for precision I never get any False positives. I think I just have a disconnect in logic between your example and how it relates to the equation for precision. $\endgroup$ – IsmailE Mar 29 at 0:25
  • 1
    $\begingroup$ @IsmailE the equation for precision (or any other measure) only says how to calculate the final performance based on the count of True Positive and False Positive cases, it doesn't say how you're supposed to count them. I don't see which point confuses you: are you clear about defining one class as positive and the other as negative? For example if sick=positive and healthy=negative, then a false positive happens when the test patient is truly healthy but the method predicts that they are sick. Note that you should build the confusion matrix with the aggregated results across all the ... $\endgroup$ – Erwan Mar 29 at 10:49
  • 1
    $\begingroup$ ... patients, this way it can contain any of the 4 possible cases. $\endgroup$ – Erwan Mar 29 at 10:50
  • $\begingroup$ Okay I see what you're saying i was still visualizing in my head that my confusion matrix can only have two cases. So in practice, I could just keep a list of my predictions and true values for each test set. Then after testing for each patient, combine all sets of predictions and true values together and produce one confusion matrix at the end containing the combined predictions and true values from all splits then correct? @Erwan $\endgroup$ – IsmailE Mar 29 at 10:57
  • 1
    $\begingroup$ @IsmailE yes, exactly. $\endgroup$ – Erwan Mar 29 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.