Stability of value function approximation in policy gradients

Question

In DQNs, function approximation of the Q-values is unstable for correlated updates. In policy gradients with a baseline, will the value function of the policy not be plagued by the same correlated updates?

For example, in the REINFORCE with baseline algorithm the updates are applied to each time step in a temporally sequential order. I understand that in policy gradients the goal is to estimate the value of the policy and not necessarily the entire state space; however, in a stochastic environment and/or under a stochastic policy, not all states will be sampled with the same probability leading to overfitting to a specific trajectory meaning the value function will not be useful as a baseline for the other trajectories of the policy. Are there algorithms that shuffle the trajectory before fitting the data and/or collect batches of trajectories and then randomly sample from the batch as in DQNs?

Answer 1

The short answer is yes. Take a look at Asynchronous Methods for RL. In a similar way that sampling from experience replay helps breaking the correlations, asynchronous methods are based on having multiple agents interacting with multiple instances of the same environment.

The Process: Each agent (called worker) collects its own experience up to a specified timestep $t_{max}$, which is stored in a batch. Then a master network performs a training update by using this batch. After the update each worker resets to an identical network to the master and they start the task all over again. The update can be synchronous (as I describe here using a batch) or asynchronous by training multiple agents with their own parameters and then update asynchronously the master network.

Why this is helpful? The experience of each agent is independent of the experience of the others and thus more diverse which helps keeping correlation down.

A couple of additional comments on your question:

• The goal in Policy Gradients (PG) is to optimize the expected reward w.r.t some parameters which is analogous to the estimation of the gradient of the policy function (parametrized by these parameters).
• In RL, and especially in Policy Gradients, you are dealing mainly with 3 things: state sample correlation, bias and variance of your gradient estimator. The introduction of the baseline is to reduce variance in the gradient estimator without introducing any bias. The baseline is called control variate and is used for reducing variance in Monte Carlo estimators (our gradient estimator here) and has zero mean.