I am trying to estimate the predictive uncertainty for a deep neural network. While I do have a labeled training set, I´m trying to measure uncertainty for some unlabeled production data.
This paper proposes the use of Deep Ensembles and Adversarial Training to compute a measurement of uncertainty. However, it uses Brier Score as a metric which requires me to know the real label of my production data. Is there a similar way or metric which does not require labeled data?
Another approach was described by Yarin Gal utilizing Monte Carlo Dropout. However I can´t get any useful results using that technique on my model.
I want to use my model in an online learning task. As the given data may very over time, I need to detect examples where my model is highly uncertain, so I can manually classify those examples.