Allow me to clarify my current understanding:
Interpreting a binary classification prediction made by a deterministic neural network
On one hand, point estimates fall on a sigmoid curve (between 0-1, where 0.5 is the classification threshold). So they already have a pseudo probability/ strength associated with them given how far away the estimate is from 0.5 (e.g. a prediction of 0.85 is much more certain than a prediction of 0.54)
Interpreting a binary classification prediction made by a bayesian (probabilistic) neural network
On the other hand, a BNN not only provides a point estimate (mean, also on a sigmoid curve) but also provides confidence intervals. So could I say something like, "here we see a sigmoid value of 0.6, AND we are X% certain that it is above 0.5?"
Is my understanding correct/ missing anything important?