I've created a model using randomForest for the following dataset: https://archive.ics.uci.edu/ml/datasets/Contraceptive+Method+Choice
The thing i'm questioning is that the results of the model when used on my training and testing sets are vastly different.
library(randomForest) library(caret) df <- read.csv('cmc.csv')
Changing values to factors
df$Wife.s.education <- as.factor(df$Wife.s.education) df$Husband.s.education <- as.factor(df$Husband.s.education) df$Wife.s.religion <- as.factor(df$Wife.s.religion) df$Wife.s.now.working. <- as.factor(df$Wife.s.now.working.) df$Husband.s.occupation <- as.factor(df$Husband.s.occupation) df$Standard.of.living.index <- as.factor(df$Standard.of.living.index) df$Media.exposure <- as.factor(df$Media.exposure) #add string representation for readiblilty df[df$Contraceptive.method.used == 1,]$Contraceptive.method.used <- "No-use" df[df$Contraceptive.method.used == 2,]$Contraceptive.method.used <- "Long-term" df[df$Contraceptive.method.used == 3,]$Contraceptive.method.used <- "Short-term" df$Contraceptive.method.used <- as.factor(df$Contraceptive.method.used)
set.seed(47) ind <- sample(2, nrow(df), replace = TRUE, prob = c(0.7,0.3)) inTrain <- createDataPartition(y=df$Contraceptive.method.used, p=.7, list = FALSE) training = df[ind==1,] testing = df[ind==2,]
model <- randomForest(Contraceptive.method.used ~., data = training, proximity=TRUE)
#Prediction & Confusion Matrix - training data p1 <- predict(model, training) confusionMatrix(p1, training$Contraceptive.method.used)
Confusion Matrix Results (training):
Reference Prediction Long-term No-use Short-term Long-term 208 9 18 No-use 4 408 5 Short-term 16 14 338 Overall Statistics Accuracy : 0.9353 95% CI : (0.9184, 0.9496) No Information Rate : 0.4225 P-Value [Acc > NIR] : < 2e-16 Kappa : 0.9002 Mcnemar's Test P-Value : 0.09773
#Prediction & Confusion Matrix - testing data p2 <- predict(model, testing) confusionMatrix(p2, testing$Contraceptive.method.used)
Confusion Matrix Results (testing)
Reference Prediction Long-term No-use Short-term Long-term 42 11 23 No-use 27 122 47 Short-term 36 65 80 Overall Statistics Accuracy : 0.5386 95% CI : (0.4915, 0.5853) No Information Rate : 0.4371 P-Value [Acc > NIR] : 8.972e-06 Kappa : 0.2788 Mcnemar's Test P-Value : 0.005869
As we can see theres is a massive change in the two results, if i print my model I get the following results:
Type of random forest: classification Number of trees: 500 No. of variables tried at each split: 3 OOB estimate of error rate: 46.96% Confusion matrix: Long-term No-use Short-term class.error Long-term 76 63 89 0.6666667 No-use 45 282 104 0.3457077 Short-term 72 106 183 0.4930748
This makes me belief that the test results are correct however I'm not sure why there is such a big difference when used on the training set, is this due to overfitting? If so how can this be handled?
Any guidance would be awesome.