I am starting to understand Reinforcement Learning playing with the GYM Cart-pole environment. I would like to ask if the experience replay can slow down the training and if there is a maximum of the batch size?

Maybe I misunderstood, and when the program starts the fit function on the replay I guess it is not sending the action to the agent, but the environment is still running. Not sure if I missed a point or if the fit function run so quickly that the agent will have plenty of time to catch up with the environment for the next action during training.


Definitely using experience replay can slow down the agent processing each time step, because typically on each time step, a result is stored (possibly requiring another result to be deleted), a mini-batch number of results are chosen randomly and fetched, then the function approximator (usually a NN) has to be run forward for the mini-batch to discover max Q values (potentially multiple times per item if the NN uses action as input instead of multiple action values as output). Then it has to run a forward pass to calculate error values and back propagation and weight updates over the mini-batch. All that can take considerable amount of CPU and memory bandwidth compared to the same system running without experience replay.

However, there are a lot of factors to consider whether this is important. Here are a few that I can think of:

  • Typically the environments using DQN agents are not training in real time. Often they run multiple times the speed of real time in order to collect data quickly and solve the environment faster.

  • Obviously something without the overhead of experience replay could collect more data in the same real time if the environment is simulated, but actually the stabilising effect of experience replay is more important and the effective "wall clock" speed of learning is faster and more reliable, using less game play experience.

  • Due to the way mini-batch processing is parallelised for neural networks on GPUs, there is not such a high overhead for increasing mini-batch size as you may think. At least up to the limits of GPU. There's still a cost for memory transfers etc per item, so it's not completely free.

when the program starts the fit function on the replay I guess it is not sending the action to the agent,

That is correct, the agent is updating its internal estimates of values in order to learn the optimal control actions for later actions. The mini-batch actions are not taken, just used to refine the estimates.

but the environment is still running.

That is only correct in real-time systems. The Gym environments are not real-time as far as I know. They all run at the speed that the agent wants them to - they are effectively a turn-based game for the agent to play, although many of them are based on systems that run at a specific real-time rate. However, real-time reinforcement learning and control systems are a thing in other domains. If learning must be done online in such a system, then typically you have some choices on how to design the system:

  • Have fixed time steps, and allow for maximum computation time between each one (with perhaps a margin of error).

  • Asynchronous interactions with the environment - with fixed time steps for sampling observations and taking actions - and learning continuously sampling from recent history and updating value functions and/or policy.

  • Allow for variable length time steps. I don't know much about this approach, but it is an option.

Which to use depends on the problem, and a bit of guesswork.

  • $\begingroup$ Dear @NeilSlater, Thank you very much this was really helpful!!! $\endgroup$ Jul 14 '18 at 10:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.