2
$\begingroup$

I am working on a multi-label classification NN using genomic data. there are 10 samples and 2 ground truth labels (age and gender) for every sample. I use a sigmoid activation at the final layer and since samples are no longer constrained to the probability distribution across classes, I get samples that might have a probability of 0.5 or above for multiple ages - say age1 and age2. Since this is genomic data, this is not surprising, especially with the distribution of ages.

I am generating confusion matrices as well as precision and recall reports using sklearn

from sklearn.metrics import classification_report, multilabel_confusion_matrix

preds_train = model.predict(X_train)
preds_test = model.predict(X_test)

preds_train = preds_train > 0.5
preds_test = preds_test > 0.5

preds_train = tf.cast(preds_train, tf.float32)#one-hot predictions
preds_test = tf.cast(preds_test, tf.float32)

print(classification_report(y_true_train, preds_train))
print(classification_report(y_true_test, preds_test))

multilabel_confusion_matrix(targets_train, np.array(preds_train))

As mentioned, I get a scenario in which some samples have probabilities of more than 0.5 for the correct and also for the 'incorrect' class producing a one-hot vector containing three 1's for that sample as opposed to two. This means these samples get a turn at being true positive when evaluating precision-recall and F1 for that class as well as being a false positive for the incorrect class.

My question is, in principle is this ok and valid? Apart from increasing the threshold is there anything else that can be done?

I would be inclined to say that precision and recall and F1 are still correct, as it doesn't matter if a sample is TP and FP if this is just the pattern that is present and represents what future predictions might show!

I appreciate any input. Thank you!

$\endgroup$
0
$\begingroup$

If I understand correctly in this specific case I agree with you, it's reasonable to count such a sample as both TP and FP. However this must be explained clearly when you describe the evaluation method, since it's not the standard behaviour of precision/recall.

The alternative more standard version would be to apply a true multi-label evaluation, i.e. evaluate precision and recall for each label independently. In this setting you would obtain precision/recall/f-score values for each label, e.g. age1 and age2. This can be useful to observe the detailed performance by label, and you could still combine them into a global performance measure with micro and/or macro f-score. I would expect that it shouldn't be very different from your variant but I'm not sure. In case they differ it might be worth providing both evaluation methods.


[edit] Finally I should mention an intermediate option which consists in counting each label as a proportion of the instance: if there are $n$ labels, each label counts for $1/n$ of its classification status. For example an instance can be 1/3 TP, 1/3 FP and 1/3 TN.

$\endgroup$
8
  • $\begingroup$ Hi, thank you for your answer. I agree it would be clearly explained. so just to make this more concrete - i have an example say of 3 probabilities as follows [0.1, 0.8, 0.6] - this is a binary classification so all are independent. the correct class is class 1 (0.8) but as you can see it will be FP for 3 (0.6). because of the threshold. this is what i was unsure of. can you expand on the standard version - my initial thought was that for multi-label cases (and for sklearns implementation), that each label is actually being treated independently (this is what i thought was the case) $\endgroup$ Jul 22 at 22:06
  • $\begingroup$ @user9317212 yes, you're correct: each label is independent in a multi-label setting, which means that if there are 3 labels it's equivalent to 3 independent binary classifiers. The predicted probability represents the likelihood of the instance having this label as opposed as not having it. In your example the instance would be predicted as negative for the first label, then positive for the 2nd and 3rd label. This is why the standard evaluation requires treating each label independently (what I described in the 2nd paragraph). $\endgroup$
    – Erwan
    Jul 22 at 22:27
  • 1
    $\begingroup$ Note that your special evaluation setting works only if a small minority of cases fall in this case, because in general one could end up with weird cases where the number of FP is higher than the number of instances for example (if several labels are predicted positive). Actually this makes me think of another method, I'm going to add it to the answer. $\endgroup$
    – Erwan
    Jul 22 at 22:32
  • $\begingroup$ but even when treating the labels independently the second class will be a TP but the third will inevitably be a FP - thats simply because of the probability from the model - so when evaluating class two - the example I gave for that 1 sample [0.1, 0.8, 0.6] - when independently evaluating each class.. class 1 will be TP for this sample because that is the ground truth, but when evaluating class 2 independently, this will sample will be a FP due to the second probability (0.6) - so this is what I need clarification on, I imagine this is still correct as that is simply the models prediction? $\endgroup$ Jul 22 at 22:39
  • $\begingroup$ @user9317212 yes that's correct, but the different in the standard evaluation is that you don't calculate precision/recall/f-score globally, each label has its own score. This way a TP for one label doesn't get mixed with a FP for the other label, they are counted completely separately. It's really like evaluating 3 different classifiers independently. $\endgroup$
    – Erwan
    Jul 22 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.