# Improving classifier performances in R for imbalanced dataset

I have used an "adabag"(boosting + bagging) model on an imbalanced dataset (6% positive), I have tried to maximized the sensitivity while keeping the accuracy above 70% and the best results I got were:

• ROC= 0.711
• SENS=0.94
• SPEC=0.21

The results aren't Inhofe, especially the bad specificity. Any suggestion on how to improve the result? Can the optimization be improved, or would the addition of a penalty term help?

This is the code:

ctrl <- trainControl(method = "cv",
number = 5,
repeats = 2,
p = 0.80,
search = "grid",
initialWindow = NULL,
horizon = 1,
fixedWindow = TRUE,
skip = 0,
verboseIter = FALSE,
returnData = TRUE,
returnResamp = "final",
savePredictions = "all",
classProbs = TRUE,
summaryFunction = twoClassSummary,
preProcOptions = list(thresh = 0.80, ICAcomp = 3, k = 7, freqCut = 90/10,uniqueCut = 10, cutoff = 0.2),
sampling = "smote",
selectionFunction = "best",
index = NULL,
indexOut = NULL,
indexFinal = NULL,
timingSamps = 0,
predictionBounds = rep(FALSE, 2),
seeds = NA,
adaptive = list(min = 5,alpha = 0.05, method = "gls", complete = TRUE),
trim = FALSE,
allowParallel = TRUE)

grid <- expand.grid(maxdepth = 25, mfinal = 4000)

classifier <- train(x = training_set[,-1],y = training_set[,1], method = 'AdaBag',trControl = ctrl,metric = "ROC",tuneGrid = grid)
prediction <- predict(classifier, newdata= test_set,'prob')


plot from classifierplots package:

I tried xgboost as well.

Here is the code:

gbmGrid <- expand.grid(nrounds = 50, eta = 0.3,max_depth = 3,gamma = 0,colsample_bytree=0.6,min_child_weight=1,subsample=0.75)

ctrl <- trainControl(method = "cv",
number = 10,
repeats = 2,
p = 0.80,
search = "grid",
initialWindow = NULL,
horizon = 1,
fixedWindow = TRUE,
skip = 0,
verboseIter = FALSE,
returnData = TRUE,
returnResamp = "final",
savePredictions = "all",
classProbs = TRUE,
summaryFunction = twoClassSummary,
sampling = "smote",
selectionFunction = "best",
index = NULL,
indexOut = NULL,
indexFinal = NULL,
timingSamps = 0,
predictionBounds = rep(FALSE, 2),
seeds = NA,
adaptive = list(min = 5,alpha = 0.05, method = "gls", complete = TRUE),
trim = FALSE,
allowParallel = TRUE)

classifier <- train(x = training_set[,-1],y = training_set[,1], method = 'xgbTree',metric = "ROC",trControl = ctrl,tuneGrid = gbmGrid)
prediction <- predict(classifier, newdata= test_set[,-1],'prob')


plot from classifierplots package:

Update:

I tried asymmetric adaboost, this is the code:

model_weights <- ifelse(training_set$readmmited == "yes", (1/table(training_set$readmmited)[1]) * 0.4,

ctrl <- trainControl(method = "repeatedcv",
number = 5,
repeats = 2,
search = "grid",
returnData = TRUE,
returnResamp = "final",
savePredictions = "all",
classProbs = TRUE,
summaryFunction = twoClassSummary,
selectionFunction = "best",
allowParallel = TRUE)

classifier <- train(x = training_set[,-1],y = training_set[,1], method = 'ada',trControl = ctrl,metric = "ROC",weights = model_weights)


but the specificity is zero, what am I doing wrong?

• I think asymmetric Adaboost should do the trick. Please try that once and see the scores. Apr 12 '17 at 15:47
• @Rahul do you know how to interrupt the plots from classifierplots? Apr 12 '17 at 15:55
• I'm not sure what you mean by that can you elaborate? Apr 13 '17 at 0:08
• @Rahul why asymmetric Adaboost? do you see something in the classifierplots to suggest that? Apr 13 '17 at 7:25
• The fact that you were using XG boost gave me the idea. Also, you've mentioned that it was an imbalanced dataset. Asymmetric Adaboost works particularly well with Imbalanced datasets which aren't artificially supersampled. It is able to have higher accuracy because you have to give a certain weight to a sample. This helps overcome the imbalanced dataset disadvantage. Apr 13 '17 at 11:02