Difference between advantages of Experience Replay in DQN2013 paper

Question

I've been re-reading the Playing Atari with Deep Reinforcement Learning (2013) paper. It lists three advantages of experience replay:

This approach has several advantages over standard online Q-learning [23]. First, each step of experience is potentially used in many weight updates, which allows for greater data efficiency. Second, learning directly from consecutive samples is inefficient, due to the strong correlations between the samples; randomizing the samples breaks these correlations and therefore reduces the variance of the updates. Third, when learning on-policy the current parameters determine the next data sample that the parameters are trained on. For example, if the maximizing action is to move left then the training samples will be dominated by samples from the left-hand side; if the maximizing action then switches to the right then the training distribution will also switch. It is easy to see how unwanted feedback loops may arise and the parameters could get stuck in a poor local minimum, or even diverge catastrophically [25]. By using experience replay the behavior distribution is averaged over many of its previous states, smoothing out learning and avoiding oscillations or divergence in the parameters. Note that when learning by experience replay, it is necessary to learn off-policy (because our current parameters are different to those used to generate the sample), which motivates the choice of Q-learning.

I am confused on how the second and third advantages differ. Isn't the third advantage just another case of breaking correlation?

Thank you in advance for your help!

Answer 1

Isn't the third advantage just another case of breaking correlation?

It could be viewed that way, but it is a different kind of correlation.

The second advantage is about breaking correlation due to samples being from adjacent time steps on the same trajectory. This is a more important problem when state vectors evolve slowly/incrementally per time step.

The third advantage is about breaking correlation due to samples being taken from the same policy. This is a more important problem when some actions have very similar reward and state progression independently of state, or may result in no change to state (e.g. an agent tries to move into a wall - agents getting stuck in corners due to runaway feedback of the direction being "best" is a thing you can sometimes observe whilst DQN is learning).

Both can also be a factor when an early event puts the remainder of a trajectory into a single part of the overall space, which can occur in environments where state history is important - in those cases, almost all of the time steps in a single episode could be correlated. Think of a resource management game, where expending or keeping a key resource early on in the game has a large influence on eventual success at a task. This is impacted by elements of second and third advantages - i.e. the states in the trajectory are correlated due to state of the resource throughout, and across multiple episodes the current policy might prefer expending or keeping that resource at a specific stage.

The correlations avoided by the second and third advantages typically occur over different scales of time, although that does depend strongly on the specific problem. The second advantage might be gained with a relatively small replay memory (depending on how the state evolves and lengths of episodes). The third advantage will usually require a larger memory, in order that it captures episodes with varying policies. In addition the replay memory should include exploratory actions which also help with this third issue.