Big difference in randomTree accuracy with train and test sets

Question

I've created a model using randomForest for the following dataset: https://archive.ics.uci.edu/ml/datasets/Contraceptive+Method+Choice

The thing i'm questioning is that the results of the model when used on my training and testing sets are vastly different.

library(randomForest)
library(caret)
df <- read.csv('cmc.csv')

Changing values to factors

df$Wife.s.education <- as.factor(df$Wife.s.education)
df$Husband.s.education <- as.factor(df$Husband.s.education)
df$Wife.s.religion <- as.factor(df$Wife.s.religion)
df$Wife.s.now.working. <- as.factor(df$Wife.s.now.working.)
df$Husband.s.occupation <- as.factor(df$Husband.s.occupation)
df$Standard.of.living.index <- as.factor(df$Standard.of.living.index)
df$Media.exposure <- as.factor(df$Media.exposure)
#add string representation for readiblilty 
df[df$Contraceptive.method.used == 1,]$Contraceptive.method.used <- "No-use"
df[df$Contraceptive.method.used == 2,]$Contraceptive.method.used <- "Long-term"
df[df$Contraceptive.method.used == 3,]$Contraceptive.method.used <- "Short-term"
df$Contraceptive.method.used <- as.factor(df$Contraceptive.method.used)

Splitting data:

set.seed(47)
ind <- sample(2, nrow(df), replace = TRUE, prob = c(0.7,0.3))
inTrain <- createDataPartition(y=df$Contraceptive.method.used, p=.7, list = FALSE)
training = df[ind==1,]
testing = df[ind==2,]

Creating model:

model <- randomForest(Contraceptive.method.used ~., data = training, proximity=TRUE)

#Prediction & Confusion Matrix - training data
p1 <- predict(model, training)
confusionMatrix(p1, training$Contraceptive.method.used)

Confusion Matrix Results (training):

          Reference
Prediction   Long-term No-use Short-term
  Long-term        208      9         18
  No-use             4    408          5
  Short-term        16     14        338

Overall Statistics

               Accuracy : 0.9353          
                 95% CI : (0.9184, 0.9496)
    No Information Rate : 0.4225          
    P-Value [Acc > NIR] : < 2e-16         

                  Kappa : 0.9002          

 Mcnemar's Test P-Value : 0.09773    

#Prediction & Confusion Matrix - testing data
p2 <- predict(model, testing)
confusionMatrix(p2, testing$Contraceptive.method.used)

Confusion Matrix Results (testing)

    Reference
Prediction   Long-term No-use Short-term
  Long-term         42     11         23
  No-use            27    122         47
  Short-term        36     65         80

Overall Statistics

               Accuracy : 0.5386          
                 95% CI : (0.4915, 0.5853)
    No Information Rate : 0.4371          
    P-Value [Acc > NIR] : 8.972e-06       

                  Kappa : 0.2788          

 Mcnemar's Test P-Value : 0.005869  

As we can see theres is a massive change in the two results, if i print my model I get the following results:

     Type of random forest: classification
                     Number of trees: 500
No. of variables tried at each split: 3

        OOB estimate of  error rate: 46.96%
Confusion matrix:
           Long-term No-use Short-term class.error
Long-term         76     63         89   0.6666667
No-use            45    282        104   0.3457077
Short-term        72    106        183   0.4930748

This makes me belief that the test results are correct however I'm not sure why there is such a big difference when used on the training set, is this due to overfitting? If so how can this be handled?

Any guidance would be awesome.

Answer 1

Overfitting is a common problem in random forest, you can use cross Validation methods (Cross validation is a technique to build models that are not prone to overfitting) example - K-fold cross validation, stratified k fold..

Try this using train package-
control <- trainControl(method = "cv", number = 5) and add this as a control parameter to the model.

Answer 2

You are right. Its looks like typical overfitting. How to handle it?

• try different test:train ratio
• K-Fold Cross-Validation
• use feature selection techniques
• Early stopping
• Regularization

Answer 3

Before training and testing any Machine Learning model, we should perform some prerequisite steps to tune the accuracy of model and to avoid Overfitting or Underfitting.

• First you should understand the data.

• Analyse statistics of data like range, min, max, std dev etc.

• Visualize the data as it will give you better insight of data.

• If each features in data has different value range, then perform data normalization and scaling.

• You must select relevant features and avoid unrelated features.

• Convert categorical data into numeric.

• Data may contain unknown or missing data. You must handle missing data appropriately.

• Outliers detection and handling of such data is very important.

• There might be some situation where you need to derive some more features from available features.

After all that preprocessing of data, you have to split data into Train-Test data. For that there are different different strategies.

When you are designing ML model, there are some hyperparameters which you need to tune before training model. These parameters control the performance and accuracy of model. Hyperparameters are like number of trees in Random Forest algorithm. Depth and width of each tree etc.