# Over-fitting issue in a classification problem (unbalanced data)

I am working on a rare event (unbalanced target variable) classification problem using decision trees. My dataset comprises of 95% non-event and 5% minority (events) class.

I used decision tree over logistic regression because I had many categorical variables comparing to continuous variables. I get a good performance for training data with the decision tree C5.0. However I get poor results for the new data. I use the confusion matrix as a measure of performance. Training model is over-fitting.

I did pruning to reduce the over-fitting caused by the decision tree. I used the following code to build the model

Classifi_C5.0 <- C5.0(TARGET ~., , data = training_data_SMOTED, trails = 500,
control = C5.0Control(minCases = mincases_count,
noGlobalPruning = FALSE))


I balanced the minority and majority class using the following code:

training_data_SMOTED <- SMOTE(TARGET ~ ., training_data,
perc.over = 100, k = 5, perc.under = 200)


• You can try cross-validation for fitting your hyper-parameters. It might not help that much in your case.
– mic
Jun 17 '15 at 12:10
• When you say good performance for training data, what do you mean?Predicting the majority class all the time will get you 95% accuracy. But you may have a cost matrix which makes this an unattractive choice ? Jun 17 '15 at 14:00
• @image_doctor, For training data. i get close to 75% accuracy for both majority class and minortiy class. However when i bring in some new data to test for the performance, there it is not giving good results. I get accuracy close to 40% in the new data. Simply put, my model is not robust or it is over-fitting.
– Arun
Jun 17 '15 at 18:21
• @mic , thanks for the suggestion. I tried k fold cross validation for rpart. However i dont know how to use the same in C5.0 algorithm in R.
– Arun
Jun 17 '15 at 18:23
• @image_doctor, what you are referring as cost matrix? is that a technique ?
– Arun
Jun 17 '15 at 18:24

I'm working on a similar classification problem using Quinlans C5.0 and in my experience there are couple of tweaks inside that might help you.
• There are a couple of other parameters you might find useful. The minimal leaf size (minCases in C5.0Control() is something to you want to keep as big as possible when avoiding overfitting issues.
• Another one is the confidence factor (CF) which lets you control the severity of pruning of the tree, meaning lower factor levels will likely prune away the leaves which overspecify the classification.
• Finally, you might try using the fuzzyThreshold control option which lets you 'soften' sharp thresholds with an inbetween grey area.
These are all very well documented in the C5.0 vignette as well as on the informal tutorial page.