How to best estimate the coefficients of a confusion matrix in case of strong class imbalance?

Question

I have a trained binary classifier (forget about how this was trained and think of it as a magical black box) and I would like to measure its classification performance (e.g. compute a confusion matrix) on a strongly imbalanced population/test dataset. Suppose the composition (in terms of the two classes) of the population / test set is also known - say 99% of class $A$ vs 1% of $B$.

I also would like to reduce as much as possible the uncertainty in the estimate coefficients of the confusion matrix computed on the entire (large) test set under the constraint of doing predictions for a max of $N$ examples of the test set (suppose that the predictions are expensive to run).

What's the best approach to do this?

The best thing I can come up with is doing something like "stratified sampling":

1. I create from my test set (which is arbitrarily large for simplicity) a balanced sample with $N/2$ examples belonging to $A$ and $N/2$ examples belonging to $B$.
2. I run the classifier on the balance dataset, and compute the resulting confusion matrix.
3. I "normalize" the so-obtained confusion matrix so that its rows sum to 1, obtaining $\hat{\mathbf C}$.
4. To estimate what would be the confusion matrix on a large representative sample (say of $M$ items taken from my large test set) I compute

$$ \mathbf C = \left( \begin{matrix} 0.99M & 0 \\ 0 & 0.01M \end{matrix} \right) \cdot \hat{\mathbf C} $$

P.S.: To be really happy, I would like to estimate also the uncertainty of the components of the final confusion matrix $\mathbf C$. To do that, can I just compute e.g. the credible intervals of the coefficients of $\hat{\mathbf C}$ starting from the balanced sample, and "rescale" them using the definition of $\mathbf C$ given above?

Bonus track: if somebody could send me with a reference about how to do some kind of "power test" (e.g. "how large should $N$ be to "bracket" the values of $\mathbf C$ within given tight intervals?") I'd be glad to review it.

Bonus$²$ track: any tips on how to extend the estimation of the uncertainty to nonbinary classifiers (more than 2 classes) are also welcome! $\ddot\smile$

Answer 1

There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, DP, etc.

Disclaimer:

If you use python, PyCM module can help you to find and calculate these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})  

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

>>> score = cm.Kappa