Estimating class prevalence in unlabelled data after predicting labels with a binary classifier

Question

I'm looking to get an estimate of the prevalence of 1's (i.e. the rate of positive labels) in a very large dataset that I have. However, I am hoping to report this percentage as a 95% credible interval instead of as an exact estimate of rate, taking into account the model uncertainties.

These are the steps I'm hoping to perform:

1. Train a binary classifier on labelled training data.
2. Use a labelled test set to estimate the specificity and sensitivity of the classifier.
3. Use the classifier to predict the label for the unlabelled records in the dataset.
4. Obviously I could get an exact prevalence estimate by simply calculating the mean of the predicted outputs. But this is where I'm hoping to implement an approach for reporting the prevalence estimate as an interval.

So my question is: Is there a best-practice approach to doing this? I found this study which trains a binary classifier and then uses a Bayesian prevalence model to report the prevalence as a 95% confidence interval by incorporating the uncertainty associated with the model specificity and sensitivity. However, I'm having trouble understanding exactly what they did here. I'm also not finding many others who have done something similar. So, any suggestions for a reliable approach I could take to do this would be greatly appreciated.

Thanks in advance!

Answer 1

There's a domain named quantification that deals with this kind of problem. It aims to create "quantifiers" (instead of classifiers) that will focus more on estimating the prevalence of a class in a population rather than on individual classifications. An easy approach is "adjusted count" (AC), but there are other (potentially better) approaches. You can find more in this paper or this one.

Basically, the idea of AC is:

1.1) Learn a binary classifier from the train dataset

1.2) Estimate the False Positive Rate (fpr) and True Positive Rate (tpr) from the training set, using cross-validation

2) Estimate prevalence of the test set based on the observed prevalence in the test set, corrected by estimated fpr and tpr (I guess the ideal is to have different test sets with different prevalences)

That way, you can estimate the prevalence of your sample based on the fraction of predicted positives that are actually positive, and the fraction of predicted negatives that are actually positive (and I guess you can easily compute confidence intervals).

The good thing with this is that your model will be way more robust to a change of prevalence in a population, rather than if you just count the positive and negative instances. (All this is explained better in the link papers)