I am implementing policy gradient to solve the OpenAI CartPole game. I have a loss function that goes as follows:

def build_policy_gradient_loss(model):
    action_prob_placeholder = model.output
    action_onehot_placeholder = K.placeholder(shape=(OUTPUT_DIM,))
    discounted_reward_placeholder = K.placeholder(shape=(None,))

    action_prob = K.sum(action_prob_placeholder*action_onehot_placeholder, axis=1)
    loss = -K.log(action_prob)*discounted_reward_placeholder
    loss = K.sum(loss)
    adam = optimizers.Adam()
    updates = adam.get_updates(params=model.trainable_weights, 
    return K.function(inputs=[model.input,

Long story short, after completing a single episode, this loss function takes an array of states, an array of actions, and a scalar discounted reward. This is not that desirable since I would like to average the gradient of the loss function w.r.t. the parameters over many episodes. One idea I had would be to use this loss function to derive the parameter gradients for every episode in an epoch (say 10 episodes/epoch), average the gradients, and then manually update the neural network's parameters. I was hoping there was a clearer way to do this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.