Why do we normalize the discounted rewards when doing policy gradient reinforcement learning?

Question

I'm trying to understand the policy gradient approach for solving the cartpole problem. In this approach, we're expressing the gradient of the loss w.r.t each parameter of our policy as an expectation of the sum of gradients of our policy gradient for all actions in a sequence, weighted by the sum of discounted rewards in that sequence:

$$\nabla_\theta L(\theta) = E[ G(S_{0:T}, A_{0:T})\sum_{t=0}^{T}\nabla_\theta log\pi_\theta (A_t|S_t) ]$$

and we estimate it using an empirical average across all samples in an episode - which makes sense intuitively.

BUT the less intuitive part is that I saw a common practice to normalize advantages in between episodes in several implementations (and indeed it works better). So after they calculate the they wouldn't directly use the advantage, but rather would normalize it, e.g. here they do after every episode:

discounted_epr = discount_rewards(epr)
discounted_epr -= np.mean(discounted_epr)
discounted_epr /= np.std(discounted_epr)

what's the justification for that - both in theory and in intuition? It seems to me that if an episode is long and as such has large advantages, it's worth learning more from that episode than from a 3 moves episode. What am I missing?

Answer 1

In general we prefer to normalize the returns for stability purposes. If you work out the backpropagation equations you will see that the return affects the gradients. Thus, we would like to keep its values in a specific convenient range. We don't follow this practice for theoretical guarantees but for practical reasons. The same goes with clipping $Q$ value functions in Q-learning combined with NNs. Of course, there are some drawbacks with these approaches but in general the algorithm behaves better as the backpropagation does not lead your network weights to extreme values. Please take a look at this excellent post by Andrej Karpathy (I attach the part related to your question as a blockquote) which gives additional insights:

More general advantage functions. I also promised a bit more discussion of the returns. So far we have judged the goodness of every individual action based on whether or not we win the game. In a more general RL setting we would receive some reward $r_t$ at every time step. One common choice is to use a discounted reward, so the “eventual reward” in the diagram above would become $R_t=∑^∞_{k=0}γ^kr_{t+k}$, where $\gamma$ is a number between 0 and 1 called a discount factor (e.g. 0.99). The expression states that the strength with which we encourage a sampled action is the weighted sum of all rewards afterwards, but later rewards are exponentially less important. In practice it can can also be important to normalize these. For example, suppose we compute $R_t$ for all of the 20,000 actions in the batch of 100 Pong game rollouts above. One good idea is to “standardize” these returns (e.g. subtract mean, divide by standard deviation) before we plug them into backprop. This way we’re always encouraging and discouraging roughly half of the performed actions. Mathematically you can also interpret these tricks as a way of controlling the variance of the policy gradient estimator. A more in-depth exploration can be found here.

Answer 2

You might find the following references useful:

• Section 4.1.2 Advantage Normalization: They mention that normalizing the advantage is a trick useful for training. It usually results in faster learning.
• Learning values across many orders of magnitude: They give an algorithm for normalization of rewards, and give detailed experimentation on the Atari environments. The basic idea is that the rewards can vary over a large range of magnitudes, and the function approximators being used in RL (such as neural networks) are usually not invariant to the scale of the input. So normalization becomes a key step. Do check this paper for more details.