2
$\begingroup$

I'm using random forest and the out of bag error for the level of one class is very different to its test error. I'm working with a cutt-of equal to c(0.2,0.8). Here's the case:

fmla <- as.formula(paste("ex ~ ", paste(names(muestra.fullarbol[,-c(1,2,3,9,10,11,12,17,19,20,21,22,23,24,29,31,32,33,34,35,36,47)]), collapse= "+")))
> bosque <- randomForest(fmla , data=muestra.fullarbol ,mtry=12, ntree=1000  , cutoff=c(0.2,0.8),importance=TRUE)
> bosque

Call:
 randomForest(formula = fmla, data = muestra.fullarbol, mtry = 12,      ntree = 1000, cutoff = c(0.2, 0.8), importance = TRUE) 
               Type of random forest: classification
                     Number of trees: 1000
No. of variables tried at each split: 12

        OOB estimate of  error rate: 15.81%
Confusion matrix:
     No Si class.error
No 3999  1     0.00025
Si  758 42     0.94750

As we see, the out of bag error for the level "SI" is 0.94750. However If I use the test set to get a sense of the error, the result is very different:

res.arbol <- predict(bosque,test.fullarbol,type="class")
> summary(res.arbol)
  No   Si 
7761   43 
> table(res.arbol,test.fullarbol$ex)

res.arbol   No   Si
       No 6937  824
       Si    4   39
> prop.table(table(res.arbol,test.fullarbol$ex),1)

res.arbol         No         Si
       No 0.89382811 0.10617189
       Si 0.09302326 0.90697674

Now we see thae error rate for the test set in the class "Si" is very low , it's equal to 0.093 and It doesn´t make sense to me.

I guess that the cut off is working just to predict out of the sample(muestra.fullarbol), but I'm not sure. What can be the reason of that huge difference?

$\endgroup$
1
$\begingroup$

What type of sampling method did you use to split the data? I'm guessing, by looking at the ratio of classes in your OOB confusion matrix and your test set confusion matrix that it wasn't stratified, is this correct? For instance, the class ratio in the OOB data is 4000:800 whereas in the test set it is 7761:43. As a starting point, you may want to split your data using stratified sampling so that the classes are fairly represented (in a statistical sense) in both your training data and test data. The ratio of classes in the test set should be identical or almost the same as the ratio used in the training data.

I think this would be a good starting point to figuring out the problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.