I have a trained binary classifier (forget about how this was trained and think of it as a magical black box) and I would like to measure its classification performance (e.g. compute a confusion matrix) on a strongly imbalanced test dataset. 
Suppose the composition (in terms of the two classes) of the test set is also known - say 99% of class $A$ vs 1% of $B$.

I also would like to reduce as much as possible the uncertainty in the estimate coefficients of the confusion matrix computed on the entire (large) test set under the constraint of doing predictions for a max of $N$ examples of the test set (suppose that the predictions are expensive to run).


What's the best approach to do this?
 

The best thing I can come up with is doing something like "stratified sampling": 

1. I create from my test set (which is arbitrarily large for simplicity)  a balanced sample with $N/2$  examples belonging to $A$ and $N/2$ examples belonging to $B$.
2. I run the classifier on the balance dataset, and compute the resulting confusion matrix.
3. I "normalize" the so-obtained confusion matrix so that its rows sum to 1, obtaining $\hat{\mathbf C}$.
4. To estimate what would be the confusion matrix on a large representative sample (say of $M$ items taken from my large test set) I compute

$$ 
\mathbf C = 
\left(
\begin{matrix}
0.99M & 0 \\
0 & 0.01M
\end{matrix}
\right)
\cdot \hat{\mathbf C} 
$$

P.S.: To be really happy, I would like to estimate also the _uncertainty_ of the components of the final confusion matrix $\mathbf C$. 
To do that, can I just compute e.g. the credible intervals of the coefficients of $\hat{\mathbf C}$ starting from the balanced sample, and "rescale" them using the definition of $\mathbf C$ given above?

Bonus track: if somebody could send me with a reference about how to do some kind of "power test" (e.g. "how large should $N$ be to "bracket" the values of $\mathbf C$ within given tight intervals?") I'd be glad to review it. 


Bonus$²$ track: any tips on how to extend the estimation of the uncertainty to nonbinary classifiers (more than 2 classes) are also welcome! $\ddot\smile$