Apologies if this is a basic question.
I have a very unbalanced dataset in which the records are labelled by one of two classes, class1 (negative class) and class2 (positive class):
class 1: 1.5 million records
class 2: 100 thousand records
So there is essentially a 1:15 ratio between them.
Because I am dealing with time dependent observations, I have divided the train/test data in the following way:
class1: 900 thousand
class2: ~100 thousand
class1: 600 thousand
Thus having 62% of training data and 38% of test data, the test data being more recent.
I am using a Random Forest Classifier as a model, and have given a class weight of 1 to class1 and 9 to class2, to compensate for the data imbalance.
I read that to measure the quality of a scenario with unbalanced data, I should use a Precision vs Recall curve.
The problem is, none of the 25 data points from class2 are classified correctly, and thus the true_positives = 0.
In a case like this, how can I evaluate my model effectively? Should I undersample class1 from my test data? Are there more parameters besides class weights that I should try to tune?
Thank you for your time.