I've created a hybrid model by taking an existing decision engine (TRUE/FALSE output) and supplementing it with a random forest classifier (TRUE/FALSE) model. The output of the hybrid model is produced by computing an OR from the predictions of the two models.

When evaluating the performance on historical data, I can produce a ROC for the random forest by computing the class probabilities based on the votes from all of the random trees combined. I cannot do this for the existing decision engine because it does not output probabilities. For the purpose of this analysis, I need to treat it as a black box.

How can I compute an ROC for the hybrid model? I would like to do this with R. I'm imagining that for each point on the ROC of the first model, I would need to compute the lift generated by adding the existing decision engine model.

  • $\begingroup$ I'm thinking that I can define the class probability of the hybrid model as 1 if the existing decision engine outputs TRUE, and equal to the random forest class probability if the existing decision engine outputs FALSE. Then I should see a lift in the ROC of the hybrid model as I've observed in the confusion matrix for the hybrid model compared to either model alone. $\endgroup$ – Bobby May 4 '18 at 21:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.