I understand that by filtering out the instances with labels that Random Forest trees are uncertain upon with their decisions, and model these with another classifier could give a better overall result. My question is, how can I "combine" two (or more) classifiers' classification on a single unlabeled dataset?
Here's a solution using the caret package for R. A Random Forest is first trained on the data. All observations for which the probability (from the voting) is less than 99% are then passed to model 2, linear discriminant analysis. Only the probabilities from unseen resampling observations are used, since the Random Forest will otherwise fit the training data perfectly. That is what caret is needed for.
The accuracy is a little higher for the uncertain cases, but this is probably overfitting since I have tried several different models and the data set is small.
I'd like to know if this really improves the performance in your application for the out of sample test data. Are there any papers that recommend this approach? This approach seems to resemble boosting. I tried it on some of my data but could not improve the out of sample performance I got from model 1 (a Random Forest).
data(iris) library(caret) myTrainControl <- trainControl(method = "cv", number = 10, savePredictions = T, classProbs = TRUE) set.seed(4213) # To get the same resamples every time M1 <- train(y = iris$Species, x = iris[, !(names(iris) == "Species")], method = "rf", trControl = myTrainControl) M1pred <- M1$pred[M1$pred$mtry == 2, ] confusionMatrix(M1pred$pred, M1pred$obs) # Accuracy 96% # Inspect the predicted class probabilities: probs <- cbind(M1pred$setosa, M1pred$versicolor, M1pred$virginica) colnames(probs) <- c("setosa", "versicolor", "virginica") maxCol <- max.col(probs) probs <- cbind(probs, maxCol) maxProbs <- apply(probs, 1, function(x) x[x["maxCol"]]) summary(maxProbs) # Min. 1st Qu. Median Mean 3rd Qu. Max. # 0.5120 0.9805 0.9980 0.9645 1.0000 1.0000 # Let's define anything below 99% as uncertain: uncertain <- which(maxProbs < 0.99) length(uncertain) # 45 # Find rows in iris data that belong to 'uncertain' cases: uncertainIndex <- M1pred$rowIndex[uncertain] confusionMatrix(M1pred$pred[uncertain], reference = M1pred$obs[uncertain]) # M1 Performance in uncertain cases: # Reference # Prediction setosa versicolor virginica # setosa 5 0 0 # versicolor 0 17 3 # virginica 0 3 17 # # Overall Statistics # Accuracy : 0.8667 # Train new model on uncertain data only: irisUncertain <- iris[uncertainIndex, ] set.seed(4213) # To get the same resamples every time M2 <- train(y = irisUncertain$Species, x = irisUncertain[, !(names(irisUncertain) == "Species")], method = "lda", trControl = myTrainControl) M2pred <- M2$pred confusionMatrix(M2pred$pred, reference = M2pred$obs) # Reference # Prediction setosa versicolor virginica # setosa 4 0 0 # versicolor 1 16 0 # virginica 0 4 20 # # Overall Statistics # Accuracy : 0.8889 # (Small discrepancy: Why does caret report an accuracy of 88,5% for M2?) # For new data predictions can be made as follows: # (just as an example from the original data again) # Some 'uncertain' cases are 'certain' now using the full M2 model newdat <- irisUncertain[30:34, -5] M1maxProbs <- apply(predict(M1, newdat, type = "prob"), 1, max) ifelse(M1maxProbs < 0.99, paste("M2:", predict(M2, newdat)), paste("M1:", predict(M1, newdat))) # 85 94 107 126 # "M2: versicolor" "M1: versicolor" "M2: virginica" "M1: virginica" # 71 # "M2: virginica"