# Understanding ROCs in imbalanced data-sets

A response variable (label) $B$ can either be $0$ or $1$.

In the training set, $B_i = 1$ is an extremely rare event at only $0.26\%$ occurrences. Which makes the prediction of this label on a test data-set a difficult problem.

• I used SMOTE to sample from the training-set of some $1.55 \times 10^5$ rows to obtain a completely balanced set of $620$ rows.

balanced.df <- SMOTE(B ~ ., df, perc.over = 100, perc.under = 200)

• randomForest with $1000$ trees was used to fit the model, as shown:

randomForest.Fit <- randomForest(B ~ ., data = balanced.df, ntree = 1000)

• For making a validation set, $2000$ rows were sampled at random without repetition from the data-set.

The actual frequencies of $B_i$ in the validation set are:

   0    1
1998   2


And those in the predicted set are:

   0    1
1836  164


The results seem promising, but perhaps a little too much. Also, it is essential that the percentage of False Positives are reduced.

My questions are:

• How severely do you think the skew in data is affecting the validation results?

• Is there a point in validating again by creating an arbitrary data-set with bias, for example by selecting more $B_i = 1$ in the validation set?

• What other metrics/validation techniques which reflect the "accuracy*" of prediction?

*The term accuracy is used in a generic sense.

• I think you should first split the training / test sets and then use SMOTE on both sets, otherwise you're leaking training data into the test set, which results in too optimistic scores. – stmax Jun 22 '16 at 14:34
• Of course! I'll try that. My assumption was that since I am randomly sampling a mere 2000 entries from 155,000 things wouldn't wrong. But now that I think of it, because I'm also using SMOTE ... – 119631 Jun 22 '16 at 14:41

So your data-set of 155000 records has 403 records where B=1, and B=0 for the remaining 154597 records.

You could try splitting your data-set into 2:1 training/test sets sampled by each class of B. After you've done this, then only for the training set use SMOTE to over-sample the records with B=1 along with under-sampling the B=0 training records to bring the class ratio to something like 4:1.

Over/under sampling for the test set is not required as it is supposed to mimic real world uncertainty to test your model's performance.

Your model's AUC will definitely get reduced since (as rightly pointed out by stmax), you've leaked test records into the training set by over sampling B=1 cases before splitting the train-test sets.