Is there any way to plot ROC curve for Ensemble hard voting classifier?

Question

I am working on a multi-class text classification problem and performing an Ensemble learning for text classification. I chose hard voting as ensemble technique. I tried to plot ROC curve for my ensemble method but it didn't work by showing the error that ROC curve can't be plotted for Hard voting ensemble metho. So my question is there any possible way by which I can plot ROC curve for my hard voting ensemble classifier in python.

Answer 1

For an ROC curve, you need the model to produce an ordering of the test points (ordered by likelihood of being in the positive class). A few approaches:

1. A hard voting classifier usually just outputs the final class prediction. This is an ordering, just with nearly all ties. That will produce an ROC curve of two straight line segments, from the square's corners to a single point representing the model's specificity and sensitivity. Not the most informative, but there it is.

2. While you probably are using a majority vote to produce the final class prediction, you can also interpret it slightly differently: produce a probability score by averaging the votes (as binary) from the base models. The final class prediction is the same as thresholding this score at 0.5. So now you have at least fewer ties, and you can produce an ROC curve that looks a little smoother. This will work best if you have lots of base models, and in fact is pretty much exactly what you do when building random forests.

3. If your base models come with probability scores, then for each threshold you can get the classifications from all of the base models, and have those vote to produce classes. Varying that threshold then produces an ROC curve that's just as smooth as any of the base models' ROC curves. The catch is whether one threshold represents the same thing for all the base models: you need all the base models to be well-calibrated for this to really make sense. You also lose insight into the potential power of setting different thresholds for each of your base models.

Answer 2

Might be late but one alternative for this and actually any classifier that does not return probabilities such as SVM is to use calibration either Platt scaling or Isotonic regression

In this way, you will have a classifier that actually returns a probability i.e you will be able to calculate ROC Curve

Reference:

https://mmuratarat.github.io/2019-10-12/probabilistic-output-of-svm#:~:text=SVMs%20don't%20output%20probabilities,the%20output%20to%20class%20probabilities.&text=In%20the%20binary%20case%2C%20the,validation%20on%20the%20training%20data