Starting from Yarin Gal's research paper on using Dropout as a Bayesian Approximation (https://arxiv.org/pdf/1506.02142.pdf), I am trying to apply this concept to my Sequence Prediction model. My model comprises of 2 LSTM layers, followed by a relu dense layer, followed by a softmax layer. A dropout layer is added after each of the layers.

I've found some implementations for measuring the uncertainty of a deep neural network (like this one here: https://fairyonice.github.io/Measure-the-uncertainty-in-deep-learning-models-using-dropout.html), but all of them seem to be applicable to dense layers rather than LSTM layers.

So in summary, what I am asking about is:

  1. How can I turn on Dropout during testing for LSTM layers?
  2. How can I measure my model's uncertainty?

Firstly, I suggest looking at the following paper:

Gal and Ghahramani, A Theoretically Grounded Application of Dropout in Recurrent Neural Networks, 2016

which discusses how to appropriately apply dropout as an approximate variational Bayesian model. The key is to use the same dropout mask at each timestep, rather than IID Bernoulli noise.

As for measuring model uncertainty, note that while dropout gives us an approximate variational Bayesian neural network, it does not give access to the variational posterior density, and so we cannot compute e.g. the entropy of the posterior distribution. But, we can trivially compute the predictive variance, which is closely related to model uncertainty. Simply run $n$ forward passes on an input to get outputs $Y_F = (y_1,\ldots,y_n)$. Then compute your measure (e.g., the covariance matrix in the regression case) on $Y_F$. For classification as your case seemingly is, you can use predictive entropy, for instance. For more measures see

Gal et al, Deep Bayesian Active Learning with Image Data, 2017


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.