# What is "Policy Collapse" and what are the causes?

I saw the term "policy collapse" on the comments of a tutorial for reinforcement learning.

I'm guessing that it's referred to as a policy collapse when the policy worsens over training due to a bad hyper-parameter, be it the learning rate, batch size, etc., but I couldn't find anything explaining it in clearly and in detail.

A web search for "policy collapse" "reinforcement learning" finds this question, a related one in stats.stackexchange.com and the comments section where you found the phrase. There are two other results on unrelated subjects where the words happen to appear next to each other. Then that's it - 5 results total from Google.

A google books ngrams search for policy collapse finds no references at all.

It is hard to prove a negative, but I think this is not a widely used term.

However, the comment does appear to be referring to a real phenomenon. That is where a reinforcement agent, instead of converging on the value functions for an optimal policy as it gains experience, actually diverges (and the parameters of the approximator will diverge too).

This can happen when using non-linear function approximators to estimate action-values. More generally, it tends to happen when you have the following traits in your problem:

• Using a function approximator, especially a non-linear one (although even linear function approximators can diverge)

• Using a bootstrap method, e.g. Temporal Difference (TD) Learning (including SARSA and Q-learning), where values are updated from the same value estimator applied to successive steps.

• Off-policy training. Attempting to learn the optimal policy whilst not behaving optimally (as in Q-Learning).

In Sutton and Barto's book this is called the "deadly triad". If you do a web search for "deadly triad" "reinforcement learning" you will find many more results. It is an ongoing area of research how best to combat the effect. In the paper that introduced the DQN model learning to play Atari games, the researchers applied two things that help stabilise against the effect:

• Experience replay, where transitions are not learned from immediately, but put into a pool from which mini-batches are sampled to train the approximator.

• Bootstrap estimates are made from a "frozen" copy of the learning network, updated every N training steps - i.e. when calculating the TD target $R + \gamma \hat{q}(S', A', \theta)$, use this old copy of the network.

From the comment section you linked, it appears even applying these things is not a guaranteed fix and takes some judgement. In that case it was increasing the mini-batch size for experience replay that helped to stabilise an agent playing a variant of the video game Pong.