Random forest with zero precision for unbalanced test data

Question

Apologies if this is a basic question.

I have a very unbalanced dataset in which the records are labelled by one of two classes, class1 (negative class) and class2 (positive class):

class 1: 1.5 million records

class 2: 100 thousand records

So there is essentially a 1:15 ratio between them.

Because I am dealing with time dependent observations, I have divided the train/test data in the following way:

train data:

class1: 900 thousand

class2: ~100 thousand

test data:
class1: 600 thousand

class2: 25

Thus having 62% of training data and 38% of test data, the test data being more recent.

I am using a Random Forest Classifier as a model, and have given a class weight of 1 to class1 and 9 to class2, to compensate for the data imbalance.

I read that to measure the quality of a scenario with unbalanced data, I should use a Precision vs Recall curve.

The problem is, none of the 25 data points from class2 are classified correctly, and thus the true_positives = 0.

In a case like this, how can I evaluate my model effectively? Should I undersample class1 from my test data? Are there more parameters besides class weights that I should try to tune?

Thank you for your time.

Answer 1

You mentioned that you're dealing with time-dependent observations, and I'm guessing that's why you chose a train-test split for evaluation. If you continue along this path, then I think you just need a larger sample of class2 in your test set.

How did you come up with 25 examples of class2 in your testing set? You should be using stratified sampling to produce your train-test sets. It looks like you started with a 60-40 train-test split with class1. If so, then your datasets should have the following distributions:

Train: 900,000 of class 1, 60,000 of class2

Test: 600,000 of class1, 40,000 of class2

Since you have a decent amount of data, you might want to train with more than 60%. An 80-20 split would probably provide a better estimate of your model's final accuracy.

---

In any case, I think you can do better than train-test splitting! There's a modified version of cross-validation designed for time-dependent data. And scikit-learn has an implementation: TimeSeriesSplit

If you go this route, then you will probably want to use different weights for your classes. Since class1 outnumbers class2 15:1, then you could assign a weight of 1 to class1 and 15 to class2 (assuming you care equally about the two classes).

---

To answer a few more of your questions:

Should I undersample class1 from my test data?

I don't think this will help since you're already adjusting for the class imbalance with sample weighting.

Are there more parameters besides class weights that I should try to tune?

You should always tune model hyperparameters :) For Random Forests, the most salient parameters are probably the number of trees, the max depth, and minimum impurity decrease.