Big difference in randomTree accuracy with train and test sets

I've created a model using randomForest for the following dataset: https://archive.ics.uci.edu/ml/datasets/Contraceptive+Method+Choice

The thing i'm questioning is that the results of the model when used on my training and testing sets are vastly different.

library(randomForest)
library(caret)


Changing values to factors

df$Wife.s.education <- as.factor(df$Wife.s.education)
df$Husband.s.education <- as.factor(df$Husband.s.education)
df$Wife.s.religion <- as.factor(df$Wife.s.religion)
df$Wife.s.now.working. <- as.factor(df$Wife.s.now.working.)
df$Husband.s.occupation <- as.factor(df$Husband.s.occupation)
df$Standard.of.living.index <- as.factor(df$Standard.of.living.index)
df$Media.exposure <- as.factor(df$Media.exposure)
df[df$Contraceptive.method.used == 1,]$Contraceptive.method.used <- "No-use"
df[df$Contraceptive.method.used == 2,]$Contraceptive.method.used <- "Long-term"
df[df$Contraceptive.method.used == 3,]$Contraceptive.method.used <- "Short-term"
df$Contraceptive.method.used <- as.factor(df$Contraceptive.method.used)


Splitting data:

set.seed(47)
ind <- sample(2, nrow(df), replace = TRUE, prob = c(0.7,0.3))
inTrain <- createDataPartition(y=df$Contraceptive.method.used, p=.7, list = FALSE) training = df[ind==1,] testing = df[ind==2,]  Creating model: model <- randomForest(Contraceptive.method.used ~., data = training, proximity=TRUE)  #Prediction & Confusion Matrix - training data p1 <- predict(model, training) confusionMatrix(p1, training$Contraceptive.method.used)


Confusion Matrix Results (training):

          Reference
Prediction   Long-term No-use Short-term
Long-term        208      9         18
No-use             4    408          5
Short-term        16     14        338

Overall Statistics

Accuracy : 0.9353
95% CI : (0.9184, 0.9496)
No Information Rate : 0.4225
P-Value [Acc > NIR] : < 2e-16

Kappa : 0.9002

Mcnemar's Test P-Value : 0.09773

#Prediction & Confusion Matrix - testing data
p2 <- predict(model, testing)
confusionMatrix(p2, testing\$Contraceptive.method.used)


Confusion Matrix Results (testing)

    Reference
Prediction   Long-term No-use Short-term
Long-term         42     11         23
No-use            27    122         47
Short-term        36     65         80

Overall Statistics

Accuracy : 0.5386
95% CI : (0.4915, 0.5853)
No Information Rate : 0.4371
P-Value [Acc > NIR] : 8.972e-06

Kappa : 0.2788

Mcnemar's Test P-Value : 0.005869


As we can see theres is a massive change in the two results, if i print my model I get the following results:

     Type of random forest: classification
Number of trees: 500
No. of variables tried at each split: 3

OOB estimate of  error rate: 46.96%
Confusion matrix:
Long-term No-use Short-term class.error
Long-term         76     63         89   0.6666667
No-use            45    282        104   0.3457077
Short-term        72    106        183   0.4930748


This makes me belief that the test results are correct however I'm not sure why there is such a big difference when used on the training set, is this due to overfitting? If so how can this be handled?

Any guidance would be awesome.

• Check max_depth of the trees in your Random Forest, too large depth has been main cause of overfitting in my experience. – Akavall Nov 21 at 19:59

Overfitting is a common problem in random forest, you can use cross Validation methods (Cross validation is a technique to build models that are not prone to overfitting) example - K-fold cross validation, stratified k fold..

Try this using train package-
control <- trainControl(method = "cv", number = 5) and add this as a control parameter to the model.

• Cross-validation doesn't directly affect the model built; it is just a more robust way of testing the performance of a model build pipeline. – Ben Reiniger Nov 22 at 1:18

You are right. Its looks like typical overfitting. How to handle it?

• try different test:train ratio
• K-Fold Cross-Validation
• use feature selection techniques
• Early stopping
• Regularization

Before training and testing any Machine Learning model, we should perform some prerequisite steps to tune the accuracy of model and to avoid Overfitting or Underfitting.

• First you should understand the data.

• Analyse statistics of data like range, min, max, std dev etc.

• Visualize the data as it will give you better insight of data.

• If each features in data has different value range, then perform data normalization and scaling.

• You must select relevant features and avoid unrelated features.

• Convert categorical data into numeric.

• Data may contain unknown or missing data. You must handle missing data appropriately.

• Outliers detection and handling of such data is very important.

• There might be some situation where you need to derive some more features from available features.

After all that preprocessing of data, you have to split data into Train-Test data. For that there are different different strategies.

When you are designing ML model, there are some hyperparameters which you need to tune before training model. These parameters control the performance and accuracy of model. Hyperparameters are like number of trees in Random Forest algorithm. Depth and width of each tree etc.

• Most of this advice seems too broad for the given question. Especially, tree models are completely unaffected by feature scaling. – Ben Reiniger Nov 22 at 1:25