What is "Policy Collapse" and what are the causes?

I saw the term "policy collapse" on the comments of a tutorial for reinforcement learning.

I'm guessing that it's referred to as a policy collapse when the policy worsens over training due to a bad hyper-parameter, be it the learning rate, batch size, etc., but I couldn't find anything explaining it in clearly and in detail.

A web search for "policy collapse" "reinforcement learning" finds this question, a related one in stats.stackexchange.com and the comments section where you found the phrase. There are two other results on unrelated subjects where the words happen to appear next to each other. Then that's it - 5 results total from Google.

A google books ngrams search for policy collapse finds no references at all.

It is hard to prove a negative, but I think this is not a widely used term.

However, the comment does appear to be referring to a real phenomenon. That is where a reinforcement agent, instead of converging on the value functions for an optimal policy as it gains experience, actually diverges (and the parameters of the approximator will diverge too).

This can happen when using non-linear function approximators to estimate action-values. More generally, it tends to happen when you have the following traits in your problem:

• Using a function approximator, especially a non-linear one (although even linear function approximators can diverge)

• Using a bootstrap method, e.g. Temporal Difference (TD) Learning (including SARSA and Q-learning), where values are updated from the same value estimator applied to successive steps.

• Off-policy training. Attempting to learn the optimal policy whilst not behaving optimally (as in Q-Learning).

In Sutton and Barto's book this is called the "deadly triad". If you do a web search for "deadly triad" "reinforcement learning" you will find many more results. It is an ongoing area of research how best to combat the effect. In the paper that introduced the DQN model learning to play Atari games, the researchers applied two things that help stabilise against the effect:

• Experience replay, where transitions are not learned from immediately, but put into a pool from which mini-batches are sampled to train the approximator.

• Bootstrap estimates are made from a "frozen" copy of the learning network, updated every N training steps - i.e. when calculating the TD target $R + \gamma \hat{q}(S', A', \theta)$, use this old copy of the network.

From the comment section you linked, it appears even applying these things is not a guaranteed fix and takes some judgement. In that case it was increasing the mini-batch size for experience replay that helped to stabilise an agent playing a variant of the video game Pong.

What is "Policy Collapse"?

I would define policy collapse as the phenomenon in which an RL agent is learning, the average reward per episode appears on average to be steadily increasing over time, everything is going well, then all of a sudden, the average reward per episode drops to a much lower value and struggles to recover.

What are the causes?

• Learning in RL is different than supervised learning
• In supervised learning, you're effectively just learning parameters in order to minimise a loss function, so given a sensible optimisation algorithm and large enough batch size, you would reasonably expect your algorithm to converge
• In RL, say you're using a policy gradient method such as PPO
• You're typically learning a value function (expected sum of discounted rewards as a function of the state), but this is difficult because:
1. The value function depends on the policy, which is changing from episode to episode, so your value function is trying to model a moving target
2. You're typically learning your value function with some kind of bootstrapping, and since your value function is also changing from episode to episode, this means your value function is trying to learn a doubly moving target, while each update is also causing the target to move in some way
• Now, you're also learning a policy as well, for example doing some kind of gradient ascent algorithm using a Monte Carlo approximation of the policy gradient
• But the environment is typically stochastic, and depending on the nature of the environment and the particular experience of the agent in any given episode, it might be very difficult to calculate a good estimate of the policy gradient (see EG Kakade and Langford, 2002 section 3.2), and a bad estimate of the policy gradient might lead to a decrease in performance
• In some cases, all it might take is a few consecutive unfortunate episodes, which happen to provide bad estimates for the gradients of the value function and policy, which stack up to change your parameters in the wrong direction to such an extent that they're no longer in the basin of attraction of the local optimum you were previously converging to
• In this case you will see a policy/performance collapse, and it might take a relatively long time before your policy is able to recover its previous levels of performance