Model assessment for EXTREMELY imbalanced dataset

Question

I am dealing with an extremely imbalanced dataset, with about 10,000 negative samples for each positive sample. I am now trying to come up with an adequate measurement of model accuracy but none seem to fit. Many places recommend the PR curve over the AUC curve for imbalanced datasets (e.g. The most informative curve for imbalance datasets) but it looks like all these recommendations are good for less imbalanced sets.

Since the denominator for precision takes into account the number of false positives, more negative samples in the dataset means smaller precision vale.

I noticed that my PR curve tends to look nice and informative as long as I keep the positive/negative ratio to around 1/10-20, but as more negative samples are taken into account the curve looks less and less like it should.

My question is whether there's a better way to assess model performance for super imbalanced datasets, or maybe I am missing something with my interpretation of the PR curve and its purpose.

Answer 1

I would only look at the precision/recall scores of the undersampled class (the positive class in your case).

Checking the performance on the oversampled class seems quite meaningless, since it is quite easy to achieve very high scores.

Then, balancing the precision/recall with a F-beta score will depend on your specific use case.