Estimating Predictive Uncertainty for unlabeled data

Question

I am trying to estimate the predictive uncertainty for a deep neural network. While I do have a labeled training set, I´m trying to measure uncertainty for some unlabeled production data.

This paper proposes the use of Deep Ensembles and Adversarial Training to compute a measurement of uncertainty. However, it uses Brier Score as a metric which requires me to know the real label of my production data. Is there a similar way or metric which does not require labeled data?

Another approach was described by Yarin Gal utilizing Monte Carlo Dropout. However I can´t get any useful results using that technique on my model.

I want to use my model in an online learning task. As the given data may very over time, I need to detect examples where my model is highly uncertain, so I can manually classify those examples.

Answer 1

See Authors comment in the thread here

So by default the dropout is only used in training time. Then in test time the dropout probability is set to 0. But there is a easy way to enable it so that it stays 'on' even during the test time. import keras

inputs = keras.Input(shape=(10,))
x = keras.layers.Dense(3)(inputs)
outputs = keras.layers.Dropout(0.5)(x, training=True)

model = keras.Model(inputs, outputs)

By setting training=True it remains active in during test time. This can be called Monte Carlo dropout or more popularly dropout at inference time. I hope it helps.