Reading the relevant paragrpah, we can gain a little more insight. It seems to come down to the idea of delayed reward. By only updating the model (i.e. the agent) every once in a while, we are inherently inducing the agent to make moves which are benficial over longer periods of time.
Here is the paragraphs from the post:
- Delayed reward — Keeping the pole in the air as long as possible means moving in ways that will be advantageous for both the present
and the future. To accomplish this we will adjust the reward value for
each observation-action pair using a function that weighs actions over
To take reward over time into account, the form of Policy Gradient we
used in the previous tutorials will need a few adjustments. The first
of which is that we now need to update our agent with more than one
experience at a time. To accomplish this, we will collect experiences
in a buffer, and then occasionally use them to update the agent all at
once. These sequences of experience are sometimes referred to as
rollouts, or experience traces. We can’t just apply these rollouts by
themselves however, we will need to ensure that the rewards are
properly adjusted by a discount factor
Intuitively this allows each action to be a little bit responsible for
not only the immediate reward, but all the rewards that followed. We
now use this modified reward as an estimation of the advantage in our
loss equation. With those changes, we are ready to solve CartPole!
The agent might otherwise want to just do the optimal move to keep the pole upright for the next milllisecond, but with this buffer, we are essentially only allowing the agent to consider larger chunks of time.
This will take longer to train, but should in theory (well, the assumption behind these rollouts...) lend itself to creating a more thoughtful agent.
In other contexts, such as a standard CNN used to classify images, this approach would make less sense as there isn't a temporal dimension to the task.
Saving up the errors and making one big update can have several other explanations in general:
- I can save compute, as we could formulate the graph to then only perform backprop after our buffer is full. For buffer size not close to 1, this would potentially halve the training time (making the gross simplication of an assumption that learning occurs at the same rate)
- It may be equivalent to using batches, in that it could help smooth the learning process. If you have a batch size of 1 sample, the error that is measures and used to update weights may be quite erratic. Using a batch has a nice averaging/smoothin effect. Saving up errors in a buffer, one could argue, may have a similar effect.
I should add, I haven't personally seen this in practice outside of reinforcement learning.