Evaluating the performance of a random forest classifier

Question

I'm using a random forest classifier (in R) to impute missing data in a dataset. Basically, I have a bunch of objects (companies) and I want to guess an attribute (size) from other attributes (capital, owning_group and state). The dependent attribute is a categorical variable (size) with 3 possible values (small|medium|large). A random forest (R package randomForest) on a set of 3 variables provide this output:

ff = size ~ capital + owning_group + state

Call:
 randomForest(formula = ff, data = df, importance = T, ntree = ntree, na.action = na.omit) 
               Type of random forest: classification
                     Number of trees: 2000
No. of variables tried at each split: 1

        OOB estimate of  error rate: 32.41%
Confusion matrix:
       large medium small class.error
large    238     17   237  0.51626016
medium    80     25   322  0.94145199
small     73     30  1320  0.07238229

  Overall Statistics

               Accuracy : 0.7297          
                 95% CI : (0.7112, 0.7476)
    No Information Rate : 0.8049          
    P-Value [Acc > NIR] : 1               

                  Kappa : 0.426           
 Mcnemar's Test P-Value : <2e-16          

Statistics by Class:

                     Class: large Class: medium Class: small
Sensitivity                0.7087       0.84211       0.7294
Specificity                0.8868       0.83981       0.8950
Pos Pred Value             0.5488       0.14988       0.9663
Neg Pred Value             0.9400       0.99373       0.4450
Prevalence                 0.1627       0.03245       0.8049
Detection Rate             0.1153       0.02733       0.5871
Detection Prevalence       0.2101       0.18232       0.6076
Balanced Accuracy          0.7977       0.84096       0.8122

I interpret this output as saying that the model has a 73% accuracy, and that the classifier makes a lot of mistakes for medium and large, but gets small mostly right. Does the P-value indicate that the model is not significant?

Assuming that this precision is OK for my context, how can I validate this model beyond these simple observations?

Answer 1

First of all, if you are trying to impute missing values with a RF model then take a look at the rfImpute() function.

Second, your data in unbalanced, which is why your classification is not good, your model is biased towards the majority class (small) and so it classifies a lot of you cases into the majority class. The issue of imbalance needs to be addressed.

Validating is done with a test set, as the results you have obtained from the model are already done using Cross-Validation (known as OOB scores).