I've created a hybrid model by taking an existing decision engine (
FALSE output) and supplementing it with a random forest classifier (
FALSE) model. The output of the hybrid model is produced by computing an
OR from the predictions of the two models.
When evaluating the performance on historical data, I can produce a ROC for the random forest by computing the class probabilities based on the votes from all of the random trees combined. I cannot do this for the existing decision engine because it does not output probabilities. For the purpose of this analysis, I need to treat it as a black box.
How can I compute an ROC for the hybrid model? I would like to do this with
R. I'm imagining that for each point on the ROC of the first model, I would need to compute the lift generated by adding the existing decision engine model.