Understanding why in deep reinforcement learning correlations in the data reduce the effectiveness

Question

From the paper

Human-level control through deep reinforcement learning, Mnih et al. Nature 2015

It says

Reinforcement learning is known to be unstable or even to diverge
when a nonlinear function approximator such as a neural network is
used to represent the action-value (also known as Q) function 20 .
This instability has several causes: the correlations present in the
sequence of observations

I am unsure how to understand this and cannot create any hypothetical examples where this can happen. What are some hypothetical scenarios, or real examples where the correlations present in the sequence disrupt the use of a 'deep learning' approximator is used?

Answer 1

One reason can be the high correlation between training data can cause bias the deep learner. In this way, if the learner gets a new data which is not correlated with the training data, it will have large error.

Answer 2

There is for example a paper Asynchronous Methods for Deep Reinforcement Learning (Mnih et. al. 2016, ICML) where they explain in the introduction that this was "previously thought" to be unstable. The results from the algorithm proposed in this paper (AC3) shows that DRL can be stable.