I'm using a random forest classifier (in R) to impute missing data in a dataset. Basically, I have a bunch of objects (companies) and I want to guess an attribute (
size) from other attributes (
state). The dependent attribute is a categorical variable (
size) with 3 possible values (small|medium|large). A random forest (R package randomForest) on a set of 3 variables provide this output:
ff = size ~ capital + owning_group + state Call: randomForest(formula = ff, data = df, importance = T, ntree = ntree, na.action = na.omit) Type of random forest: classification Number of trees: 2000 No. of variables tried at each split: 1 OOB estimate of error rate: 32.41% Confusion matrix: large medium small class.error large 238 17 237 0.51626016 medium 80 25 322 0.94145199 small 73 30 1320 0.07238229 Overall Statistics Accuracy : 0.7297 95% CI : (0.7112, 0.7476) No Information Rate : 0.8049 P-Value [Acc > NIR] : 1 Kappa : 0.426 Mcnemar's Test P-Value : <2e-16 Statistics by Class: Class: large Class: medium Class: small Sensitivity 0.7087 0.84211 0.7294 Specificity 0.8868 0.83981 0.8950 Pos Pred Value 0.5488 0.14988 0.9663 Neg Pred Value 0.9400 0.99373 0.4450 Prevalence 0.1627 0.03245 0.8049 Detection Rate 0.1153 0.02733 0.5871 Detection Prevalence 0.2101 0.18232 0.6076 Balanced Accuracy 0.7977 0.84096 0.8122
I interpret this output as saying that the model has a 73% accuracy, and that the classifier makes a lot of mistakes for
large, but gets
small mostly right. Does the P-value indicate that the model is not significant?
Assuming that this precision is OK for my context, how can I validate this model beyond these simple observations?