I'm looking to get an estimate of the prevalence of 1's (i.e. the rate of positive labels) in a very large dataset that I have. However, I am hoping to report this percentage as a 95% credible interval instead of as an exact estimate of rate, taking into account the model uncertainties.
These are the steps I'm hoping to perform:
- Train a binary classifier on labelled training data.
- Use a labelled test set to estimate the specificity and sensitivity of the classifier.
- Use the classifier to predict the label for the unlabelled records in the dataset.
- Obviously I could get an exact prevalence estimate by simply calculating the mean of the predicted outputs. But this is where I'm hoping to implement an approach for reporting the prevalence estimate as an interval.
So my question is: Is there a best-practice approach to doing this? I found this study which trains a binary classifier and then uses a Bayesian prevalence model to report the prevalence as a 95% confidence interval by incorporating the uncertainty associated with the model specificity and sensitivity. However, I'm having trouble understanding exactly what they did here. I'm also not finding many others who have done something similar. So, any suggestions for a reliable approach I could take to do this would be greatly appreciated.
Thanks in advance!