3
$\begingroup$

I am working on a rare event (unbalanced target variable) classification problem using decision trees. My dataset comprises of 95% non-event and 5% minority (events) class.

I used decision tree over logistic regression because I had many categorical variables comparing to continuous variables. I get a good performance for training data with the decision tree C5.0. However I get poor results for the new data. I use the confusion matrix as a measure of performance. Training model is over-fitting.

I did pruning to reduce the over-fitting caused by the decision tree. I used the following code to build the model

Classifi_C5.0 <- C5.0(TARGET ~., , data = training_data_SMOTED, trails = 500,
                      control = C5.0Control(minCases = mincases_count,
                                            noGlobalPruning = FALSE))

I balanced the minority and majority class using the following code:

training_data_SMOTED <- SMOTE(TARGET ~ ., training_data,
                              perc.over = 100, k = 5, perc.under = 200)

Any sort of advice will be helpful.

$\endgroup$
  • $\begingroup$ You can try cross-validation for fitting your hyper-parameters. It might not help that much in your case. $\endgroup$ – mic Jun 17 '15 at 12:10
  • 1
    $\begingroup$ When you say good performance for training data, what do you mean?Predicting the majority class all the time will get you 95% accuracy. But you may have a cost matrix which makes this an unattractive choice ? $\endgroup$ – image_doctor Jun 17 '15 at 14:00
  • 1
    $\begingroup$ @image_doctor, For training data. i get close to 75% accuracy for both majority class and minortiy class. However when i bring in some new data to test for the performance, there it is not giving good results. I get accuracy close to 40% in the new data. Simply put, my model is not robust or it is over-fitting. $\endgroup$ – Arun Jun 17 '15 at 18:21
  • $\begingroup$ @mic , thanks for the suggestion. I tried k fold cross validation for rpart. However i dont know how to use the same in C5.0 algorithm in R. $\endgroup$ – Arun Jun 17 '15 at 18:23
  • $\begingroup$ @image_doctor, what you are referring as cost matrix? is that a technique ? $\endgroup$ – Arun Jun 17 '15 at 18:24
5
$\begingroup$

I'm working on a similar classification problem using Quinlans C5.0 and in my experience there are couple of tweaks inside that might help you.

  • First, I believe using such big number (trials=500) for boosting is affecting the overfitting, since the model will repeatedly try and improve on the misclassified instances in every iteration. Try using a smaller boosting size, or even turning it off - it might help with the overfitting problem.
  • There are a couple of other parameters you might find useful. The minimal leaf size (minCases in C5.0Control() is something to you want to keep as big as possible when avoiding overfitting issues.
  • Another one is the confidence factor (CF) which lets you control the severity of pruning of the tree, meaning lower factor levels will likely prune away the leaves which overspecify the classification.
  • Finally, you might try using the fuzzyThreshold control option which lets you 'soften' sharp thresholds with an inbetween grey area.

These are all very well documented in the C5.0 vignette as well as on the informal tutorial page.

Hope this helps you a bit.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.