I am implementing policy gradient to solve the OpenAI CartPole game. I have a loss function that goes as follows:
def build_policy_gradient_loss(model): action_prob_placeholder = model.output action_onehot_placeholder = K.placeholder(shape=(OUTPUT_DIM,)) discounted_reward_placeholder = K.placeholder(shape=(None,)) action_prob = K.sum(action_prob_placeholder*action_onehot_placeholder, axis=1) loss = -K.log(action_prob)*discounted_reward_placeholder loss = K.sum(loss) adam = optimizers.Adam() updates = adam.get_updates(params=model.trainable_weights, loss=loss) return K.function(inputs=[model.input, action_onehot_placeholder, discounted_reward_placeholder], outputs=[loss], updates=updates)
Long story short, after completing a single episode, this loss function takes an array of states, an array of actions, and a scalar discounted reward. This is not that desirable since I would like to average the gradient of the loss function w.r.t. the parameters over many episodes. One idea I had would be to use this loss function to derive the parameter gradients for every episode in an epoch (say 10 episodes/epoch), average the gradients, and then manually update the neural network's parameters. I was hoping there was a clearer way to do this.