I'm following this Reinforcement Learning Tutorial. There, training data is collected during an episode. When the episode is done, the data is used to do backpropagation. However, instead of applying the calculated gradients to the model, they are stored in a buffer. The gradients of multiple episodes get summed up before they are applied.

I wonder if that is a reasonable thing to do. Usually, if the model is very good after training with the first episodes data, the gradients of the other episodes will be close to 0 (no change). If you used the tutorials code, the gradients of all episodes would be large, and the model would move away from the optimum. Am I missing something?

Edit: I tried buffering gradients with Q-Learning and compared it to directly updating the weights. It is working, but buffering is definitely worse.

  • $\begingroup$ There is a comment and answer from the author that I think touches upon this. $\endgroup$ – n1k31t4 Apr 26 '18 at 23:11

Reading the relevant paragrpah, we can gain a little more insight. It seems to come down to the idea of delayed reward. By only updating the model (i.e. the agent) every once in a while, we are inherently inducing the agent to make moves which are benficial over longer periods of time.

Here is the paragraphs from the post:

  • Delayed reward — Keeping the pole in the air as long as possible means moving in ways that will be advantageous for both the present and the future. To accomplish this we will adjust the reward value for each observation-action pair using a function that weighs actions over time.

To take reward over time into account, the form of Policy Gradient we used in the previous tutorials will need a few adjustments. The first of which is that we now need to update our agent with more than one experience at a time. To accomplish this, we will collect experiences in a buffer, and then occasionally use them to update the agent all at once. These sequences of experience are sometimes referred to as rollouts, or experience traces. We can’t just apply these rollouts by themselves however, we will need to ensure that the rewards are properly adjusted by a discount factor

Intuitively this allows each action to be a little bit responsible for not only the immediate reward, but all the rewards that followed. We now use this modified reward as an estimation of the advantage in our loss equation. With those changes, we are ready to solve CartPole!

The agent might otherwise want to just do the optimal move to keep the pole upright for the next milllisecond, but with this buffer, we are essentially only allowing the agent to consider larger chunks of time. This will take longer to train, but should in theory (well, the assumption behind these rollouts...) lend itself to creating a more thoughtful agent.

In other contexts, such as a standard CNN used to classify images, this approach would make less sense as there isn't a temporal dimension to the task.

Saving up the errors and making one big update can have several other explanations in general:

  1. I can save compute, as we could formulate the graph to then only perform backprop after our buffer is full. For buffer size not close to 1, this would potentially halve the training time (making the gross simplication of an assumption that learning occurs at the same rate)
  2. It may be equivalent to using batches, in that it could help smooth the learning process. If you have a batch size of 1 sample, the error that is measures and used to update weights may be quite erratic. Using a batch has a nice averaging/smoothin effect. Saving up errors in a buffer, one could argue, may have a similar effect.

I should add, I haven't personally seen this in practice outside of reinforcement learning.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.