Alternative methods of population-level classification

Question

In a recent study, I am generating a series of logistic regression and random forest models to predict some binary outcome. For the sake of understanding, suppose this outcome is the occurrence of diabetes in 1,000 patients at a local hospital. I know the occurrence of diabetes in this population, and can thus generate a number of classification models of this outcome using this population.

Suppose I then want to use these predicted models to project the probability of diabetes in a second population where I don't observe diabetes classification (say, N = 10,000). I can use my models generated above to predict diabetes on a person-by-person basis in this second population.

For this particular study, I'm most interested in predicting the level of diabetes at the population-level of this second population. For example, I may want to know the % of this second population that potentially has diabetes.

In the literature, it appears that most studies use some threshold (say, 50%) and sum up the # of observations with a predicted probability greater than the threshold. However, I've found other studies that have simply summed the probabilities to generate the # of, say, diabetes patients in this second population.

I'm not overly familiar with the machine learning literature, but do both of these approaches make sense? Both approaches make sense and likely lead to similar estimates with well-balanced data. However, unbalanced data may lead to distributions where the choice matters (see attached). Thoughts?

Answer 1

I think the second approach is much superior to the first approach for two main reasons.

Reason #1: The first approach forces you to set a parameter (the threshold) somewhat arbitrarily. When reporting your results, I think you would need to justify your choice of threshold. Sure, 50% seems reasonable prima facie, but what if your models happen to underestimate the probability of diabetes? Then a lower threshold might be appropriate.
This is particularly concerning if you are doing academic work, since it would allow you to "fudge" results by nudging the parameter in one direction or another. It's best to avoid arbitrary parameters when possible.

Reason #2: You've already noted that, for some distributions, the choice of method will yield different estimates. The second method will give you a more robust estimate of the target statistic.

To illustrate, suppose you have a very accurate model. Consider a case where your "second population" has five members, and your model outputs the following probabilities of diabetes for each member:

[0.49, 0.49, 0.49, 0.49, 0.49]

If you use the first method to estimate the percentage of the population that has diabetes (with a threshold of 50%), then your estimate will be 0%. This is clearly unrealistic - the probability of none of the patients having diabetes is only $0.51^5 = .0345 = 3.45\%$.

If you use the second method, then you will compute the expected number of diabetes cases as follows:

$0.49*1 + 0.49*1 + 0.49*1 + 0.49*1 + 0.49*1 = 2.45$ cases

If you express this estimate as a percentage of the population, then your estimate will (obviously) be 49%.

I suppose you could also consider this estimate flawed, since it is impossible to actually observe. This will become less of a problem for large populations (like your case with n=10,000), but if it bothers you then you could round the expected number of diabetes cases to the nearest whole number. In the contrived example above, this would yield an estimate of 40%. This is still a much better estimate than that obtained using the first method.

Hopefully that helps!