questions about the positive predictive value from a classifier output

[This is in the context of machine learning applied to drug discovery.]

Suppose you have trained a classifier with a set of chemical compounds of known active $$(A)$$ vs inactive $$(I)$$ class.
The model outputs a numerical value $$S$$ that is positively correlated with activity. Based on the distribution of $$S$$ for the two classes, you make your ROC curve, so you know the $$TPR$$ and $$FPR$$ for each $$S$$.

Now you submit $$N$$ new compounds to the classifier. For each compound $$C_i$$ you get a numerical output $$S_i$$. A medicinal chemist may ask you: 'your model classified this compound as 'active'; what is the probability that it is actually active?'.

First question: how would you calculate the $$PPV_i = P(A|S_i)$$, i.e. the probability that an individual compound is active given the observation of its $$S_i$$?

I would expect that you need an estimate of the prior probability of activity $$P(A)$$ to go from $$\{TPR(S_i),FPR(S_i)\}$$ to $$PPV$$, by the usual Bayesian formula:

$$P(A|S) = \frac {P(S|A) \cdot P(A)}{P(S|A) \cdot P(A) + P(S|I) \cdot P(I)} = \frac {TPR \cdot P(A)}{TPR \cdot P(A) + FPR \cdot (1-P(A))}$$

[Some of the machine learning tools we use output for each compound a class prediction and a probability, but the way the probability is calculated is wrong, IMO. That's why I'd like to check what you think.]

Next, you want to select $$n << N$$ compounds, maximizing the number of true actives. Normally that means selecting the $$n$$ compounds with largest $$S_i$$.
Suppose you've done that, so you have a set of $$n$$ values of $$S_i$$.
The chemist comes back and asks: 'you gave me these $$n$$ compounds to make, saying that they are probably active; but how many of them do you expect are actually active?'.

Second question: how would you calculate the expected number of true actives in the set of $$n$$ selected compounds?

I would have thought this was the sum of $$PPV_i$$ over the $$n$$ compounds. Would you agree?

These two questions are not theoretical thought experiments. A few months ago my company ran a project in collaboration with another company that claimed they could use machine learning and AI to suggest new compounds to make that were very different from the ones used in training, but still with a high probability of activity.
Apparently their strategy was to enumerate a huge combinatorial chemical space (i.e. make $$N$$ very large by creating virtual compounds), apply the classifier and pick the top scoring $$n$$ compounds.
They did that, the chemists in my company made the compounds, and they ALL turned out to be inactive.

Third question: shouldn't we actually have expected that to happen, given the way this was done?

This for me is the hardest part to understand, given that I am relatively new to machine learning and statistical modelling, so I would really need you to please correct me if I'm wrong in the way I am looking at this.

One problem that is often mentioned when expanding the set of compounds to classify is the 'domain of applicability'.
My understanding of it is that when the new compounds are very different in structure and perhaps other properties from the compounds used for training, the model may not be able to generalize, it won't be able to make reliable predictions (and the ROC curve will not apply anymore).
This seems clear enough, and in fact cross-validation, or even better external validation, should already tell how general the model can be.

Suppose however that all the new compounds are 'within' the domain of applicability (e.g. by some similarity metric).
The question is then: how do you check that the increase in $$N$$ is not accompanied by an increase in the number of false positives in the selection of $$n$$ compounds?

Concretely, imagine that you have classified $$N=10000$$ compounds, and by picking the $$n=100$$ top scoring ones, the lowest $$S_i$$ that you include in the selection corresponds to $$TPR=0.376,FPR=0.02$$. If you assume $$P(A) = 0.1$$, this gives:

$$PPV_{min} = \frac {0.376 \cdot 0.1}{0.376 \cdot 0.1 + 0.02 \cdot (1-0.1)} \approx 0.676$$

You're not happy with that, and you submit $$N=100000$$ compounds to the classifier (including the previous $$10000$$). Now by picking the $$n=100$$ top scoring ones, there are two possibilities:

• no compounds with higher scores came up, so the threshold is still the same as before, i.e. you're selecting the same $$100$$ compounds;
• more compounds with high scores came up, so the lowest $$S_i$$ that you accept is now higher and corresponds to $$TPR=0.316,FPR=0.01$$.

For the first scenario, I think one would conclude that none of the new compounds was better than the ones evaluated in the previous round.

For the second scenario, however, I wouldn't know what to make of it.
Yes, the lowest $$TPR/FPR$$ ratio is better, but how do we know that $$P(A)$$ has not got much worse? How do we test whether the larger proportion of compounds with high scores is due to the presence of more true positives rather than to the presence of more false positives?

What do you think? I hope at least some of the above makes sense...

Thanks!