1
$\begingroup$

I've created a simple LSTM network for testing

model = tf.keras.Sequential()
model.add(layers.LSTM(32, input_shape = (timesteps, data_dim), recurrent_dropout = 0.2))
model.add(layers.Dense(1))
model.compile(loss = 'mae', metrics = ['accuracy'], optimizer = tf.train.AdamOptimizer())

I am using dropout method as researched by Yarin Gal here and Lingxue Zhu here and my function for dropout looks like this:

f = K.function([model.layers[0].input, K.learning_phase()],
               [model.layers[-1].output])

after training I'm using the function above in the "predict_with_dropout"

def predict_with_dropout(x, f=f, n_iter=100):
    result = np.zeros((n_iter,))
    #print(f([x,1]))
    for iter in range(n_iter):
        result[iter] = f([x, 1])[0]

    return result
results = []


for point in test_X:
    results+= [predict_with_dropout([point])]

results_avg = np.apply_along_axis(np.mean, 1, results)
variance = np.apply_along_axis(np.var, 1, results)

This code is working as expected and as I understand it the "predict_with_dropout" function is using the f-function to re-train the LSTM model 100 times and within those 100 times it is dropping out certain cells of the model.

Is this the correct implementation of the papers or am I missing something? If it is correct - is there any way to speed this up?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.