Get the confidence interval for prediction results with LSTM

Question

I have an LSTM network and I use it to predict the load. I want to get the confidence interval for the prediction. I am not sure that I can get that or not. I have tried and search in in the different platform, however, I could not find the solution. Here is my simple model.

import pandas as pd pd.options.mode.chained_assignment = None # default='warn' import numpy as np

import tensorflow as tf
from datetime import datetime
from tensorflow import keras
from keras.models import Sequential
from keras.layers import LSTM, Dense, Dropout
from keras.layers.recurrent import LSTM
from matplotlib import pyplot as plt
from sklearn.preprocessing import StandardScaler

X_train = np.random.rand(10,5,2)
Y_train = np.random.rand(10,2)
X_test = np.random.rand(3, 5, 2)

model = Sequential()
model.add(LSTM(64, activation='relu', input_shape=(X_train.shape[1], X_train.shape[2]), return_sequences=True))
model.add(LSTM(32, activation='relu', return_sequences=False))
model.add(Dropout(0.2))
model.add(Dense(Y_train.shape[1], kernel_regularizer='l2'))
opt = keras.optimizers.Adam(learning_rate=0.001)
model.compile(optimizer=opt, loss='mse')
history = model.fit(X_train, Y_train, epochs=1, batch_size=200, validation_split=0.1,verbose=1)

prediction = model.predict(X_test) 

Answer 1

What you need is tensorflow probability. Indeed, you want to estimate a distribution and over that the interval of confidence for your prediction.

To do so, you cannot use mse loss function, but you need something that somehow compares probability distributions. One possibility is the likelihood function (better, as you want a loss to minimise the negative (log)likelihood).

Try to modify your model as follows:

import tensorflow as tf
import tensorflow_probability as tfp
tfd = tfp.distributions

model = Sequential()
model.add(LSTM(64, activation='relu', input_shape=(X_train.shape[1], X_train.shape[2]), return_sequences=True))
model.add(LSTM(32, activation='relu', return_sequences=False))
model.add(Dropout(0.2))
model.add(Dense(2, activation='relu')) # 2 as you want to predict mean and variance
model.add(tfp.layers.DistributionLambda(
    lambda t: tfd.Normal(loc=t[...,0], 
                         scale=0.01*tf.math.softplus(t[...,1])),
    name='normal_dist')) # note this

negloglik = lambda y, p_y: -p_y.log_prob(y) # note this
opt = keras.optimizers.Adam(learning_rate=0.001)
model.compile(optimizer=opt, loss=negloglik)
history = model.fit(train_dataset, epochs=1, batch_size=200, validation_split=0.1,verbose=1)

The only thing you have to carefully check is that since you are using a custom loss function, you need to pass tensors to your model, hence you have to convert your numpy arrays into tf.dataset

train_dataset = tf.data.Dataset.from_tensor_slices((X_train, Y_train))
test_dataset = tf.data.Dataset.from_tensor_slices((X_test, Y_test))

Finally, to get the Confidence Interval over predictions

mean = lambda x: x.mean().numpy().flatten() # multiply by the scaling factor if you used a scaler
sd = lambda x: x.stddev().numpy().flatten() 


def conf_int(pred):
    """95% confidence interval"""
    return np.array([mean(pred) - 2*sd(pred), mean(pred) + 2*sd(pred)]