In a recent study, I am generating a series of logistic regression and random forest models to predict some binary outcome. For the sake of understanding, suppose this outcome is the occurrence of diabetes in 1,000 patients at a local hospital. I know the occurrence of diabetes in this population, and can thus generate a number of classification models of this outcome using this population.
Suppose I then want to use these predicted models to project the probability of diabetes in a second population where I don't observe diabetes classification (say, N = 10,000). I can use my models generated above to predict diabetes on a person-by-person basis in this second population.
For this particular study, I'm most interested in predicting the level of diabetes at the population-level of this second population. For example, I may want to know the % of this second population that potentially has diabetes.
In the literature, it appears that most studies use some threshold (say, 50%) and sum up the # of observations with a predicted probability greater than the threshold. However, I've found other studies that have simply summed the probabilities to generate the # of, say, diabetes patients in this second population.
I'm not overly familiar with the machine learning literature, but do both of these approaches make sense? Both approaches make sense and likely lead to similar estimates with well-balanced data. However, unbalanced data may lead to distributions where the choice matters (see attached). Thoughts?