# LSTM: Converting to Bayesian Deep Neural Network

Starting from Yarin Gal's research paper on using Dropout as a Bayesian Approximation (https://arxiv.org/pdf/1506.02142.pdf), I am trying to apply this concept to my Sequence Prediction model. My model comprises of 2 LSTM layers, followed by a relu dense layer, followed by a softmax layer. A dropout layer is added after each of the layers.

I've found some implementations for measuring the uncertainty of a deep neural network (like this one here: https://fairyonice.github.io/Measure-the-uncertainty-in-deep-learning-models-using-dropout.html), but all of them seem to be applicable to dense layers rather than LSTM layers.

1. How can I turn on Dropout during testing for LSTM layers?
2. How can I measure my model's uncertainty?

As for measuring model uncertainty, note that while dropout gives us an approximate variational Bayesian neural network, it does not give access to the variational posterior density, and so we cannot compute e.g. the entropy of the posterior distribution. But, we can trivially compute the predictive variance, which is closely related to model uncertainty. Simply run $$n$$ forward passes on an input to get outputs $$Y_F = (y_1,\ldots,y_n)$$. Then compute your measure (e.g., the covariance matrix in the regression case) on $$Y_F$$. For classification as your case seemingly is, you can use predictive entropy, for instance. For more measures see