Understanding ROCs in imbalanced data-sets

Question

A response variable (label) $B$ can either be $0$ or $1$.

In the training set, $B_i = 1$ is an extremely rare event at only $0.26\%$ occurrences. Which makes the prediction of this label on a test data-set a difficult problem.

• I used SMOTE to sample from the training-set of some $1.55 \times 10^5$ rows to obtain a completely balanced set of $620$ rows.

  balanced.df <- SMOTE(B ~ ., df, perc.over = 100, perc.under = 200)

• randomForest with $1000$ trees was used to fit the model, as shown:

  randomForest.Fit <- randomForest(B ~ ., data = balanced.df, ntree = 1000)

• For making a validation set, $2000$ rows were sampled at random without repetition from the data-set.

The actual frequencies of $B_i$ in the validation set are:

   0    1 
 1998   2

And those in the predicted set are:

   0    1 
 1836  164

The results seem promising, but perhaps a little too much. Also, it is essential that the percentage of False Positives are reduced.

My questions are:

• How severely do you think the skew in data is affecting the validation results?

• Is there a point in validating again by creating an arbitrary data-set with bias, for example by selecting more $B_i = 1$ in the validation set?

• What other metrics/validation techniques which reflect the "accuracy*" of prediction?

*The term accuracy is used in a generic sense.

Answer 1

So your data-set of 155000 records has 403 records where B=1, and B=0 for the remaining 154597 records.

You could try splitting your data-set into 2:1 training/test sets sampled by each class of B. After you've done this, then only for the training set use SMOTE to over-sample the records with B=1 along with under-sampling the B=0 training records to bring the class ratio to something like 4:1.

Over/under sampling for the test set is not required as it is supposed to mimic real world uncertainty to test your model's performance.

Your model's AUC will definitely get reduced since (as rightly pointed out by stmax), you've leaked test records into the training set by over sampling B=1 cases before splitting the train-test sets.

The answers to each of your questions are:

1. Yes, class imbalance does effect a random forest model's accuracy. How severely depends on the severity of the imbalance as well as the nature of the data-set itself.
2. Yes, you are biasing the training by over sampling the minority class, but if these are large enough samples in the original set, then hopefully they are close representations to the entire population.
3. I would recommend two metrics/techniques you could use here: Kappa Statistic (refer to this article) and the precision-recall curves to compare different models.