0
$\begingroup$

I am implementing policy gradient to solve the OpenAI CartPole game. I have a loss function that goes as follows:

def build_policy_gradient_loss(model):
    action_prob_placeholder = model.output
    action_onehot_placeholder = K.placeholder(shape=(OUTPUT_DIM,))
    discounted_reward_placeholder = K.placeholder(shape=(None,))

    action_prob = K.sum(action_prob_placeholder*action_onehot_placeholder, axis=1)
    loss = -K.log(action_prob)*discounted_reward_placeholder
    loss = K.sum(loss)
    adam = optimizers.Adam()
    updates = adam.get_updates(params=model.trainable_weights, 
                               loss=loss)
    return K.function(inputs=[model.input,
                              action_onehot_placeholder,
                              discounted_reward_placeholder],
                      outputs=[loss],
                      updates=updates)

Long story short, after completing a single episode, this loss function takes an array of states, an array of actions, and a scalar discounted reward. This is not that desirable since I would like to average the gradient of the loss function w.r.t. the parameters over many episodes. One idea I had would be to use this loss function to derive the parameter gradients for every episode in an epoch (say 10 episodes/epoch), average the gradients, and then manually update the neural network's parameters. I was hoping there was a clearer way to do this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.