Does it make sense to build a ROC for a decision tree where there are multiple threshold you can adjust?

Question

I understand building a ROC curve when the output is a probability, say, from a logistic regression model. You can build a ROC curve by varying the cutoff threshold.

But what about decision trees of the form:

if attribute_1 > x:
  decision = positive
else:
  if attribute_2 < y:
     decision = position
  else: 
     decision = negative

You can adjust the cutoff for both attributes and all will affect your confusion matrix. Does it make sense to build a ROC curves when there are multiple thresholds?

Thanks

Answer 1

I say that is would make sense.

The ROC curve plots sensitivity and specificity. While those are functions of the thresholds, the thresholds themselves do not appear on the ROC curve.

Consequently, as you calculate sensitivity and specificity at the various values of your thresholds, you get sensitivity-specificity pairings that you can put on your graph.

I don’t have an example of either of these, but I find it plausible that such a plot could exhibit strange behavior, such as decreasing or forming a loop. This would signal to me that my approach might not make much sense, but nothing will stop you from graphing the points.

Answer 2

The ROC Curve has no relation with the way your model works, but instead, with its outputs. If the target is binary and your model outputs anything in between 0 to 1 (e.g. [0, 0.2, 0.4, ..., 1] or continuous probabilities), there's a sense in building the ROC Curve. If instead, the only outputs of your model are 0 or 1, the ROC Curve would be kind of useless, and computing simpler metrics like Precision and Recall would make more sense.