Assume we work with neural networks, with the policy gradients method. The gradient w.r.t to the objective function $$J$$, is an expectation.

In other words, to get this gradient $$\nabla_{\theta} J(\theta)$$, we sample N trajectories, then average out their gradient contribution to obtain a more precise value that can "begin flowing into our network" during backprop. $$\nabla_{\theta} J(\theta) \approx \frac1N \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_\theta \log \pi_\theta (a_{i,t}|s_{i,t}) \right) \left( \sum_{t=1}^T r(s_{i,t}, a_{i,t}) \right)$$

Looking with more detail at some specific trajectory $$i$$, we sum the gradient at each timestep in this trajectory, then multiply by the the total reward obtained from running that trajectory.

Assuming we use softmax as the network's final layer, the values for each action belongs to the range $$[0,1]$$

If I remember correctly, the gradient wrt $$\theta$$ at each timestep of a trajectory is going to be: $$\nabla_{\theta}\log \pi_{\theta}(a_{t}|s_{t}) = \frac{1}{\pi_{\theta}(a_{t}|s_{t})} \nabla \pi_{\theta}(a_{t}|s_{t})$$

where $$\nabla \pi_{\theta}(a_{t}|s_{t})$$ is then simply a derivative of softmax wrt its inputs, with the rest of the backprop towards the weight $$\theta$$

Question:

Let's say at timestep $$t_2$$ our softmax has output something like this:

[0.5,  0.1,  0.1,  0.1,  0.1,  0.1]


and the total reward for the trajectory was 50

Ignoring the rest of the backprop at the current timestep, $$\frac{1}{\pi_{\theta}(a_{t}|s_{t})}$$ will give us:

[2,  10,  10,  10,  10,  10]


This means we are already favoring other actions instead of the first action. This seems counter-intuitive to me. What if we only got such a large large reward because we took the first action? But the formula encourages us to strengthen the other 4 actions.

Yes, you are right that the term $\frac{1}{\pi_\theta(a_t|s_t)}$ itself is favoring the less likely actions. But don't forget how $\nabla \pi_\theta(a_t|s_t)$ is calculated, and you will see that $\nabla \pi_\theta(a_t|s_t)$ is over-emphasizing on the first action and $\frac{1}{\pi_\theta(a_t|s_t)}$ is correcting it.

Recall that the derivative for y =softmax(x) is $\frac{\partial y_i}{\partial x_j} = y_i(\delta_{ij}-y_i)$. Notice that $y_i$ is multiplied in the front, i.e. over-emphasizing the more likely $y_i$. By multiplying $\frac{1}{y_i}$ in the front, the over-emphasizing effect is being corrected.

In your example, when ${\pi_\theta(a_t|s_t)}=[0.5, 0.1, 0.1, 0.1, 0.1, 0.1]$, we can write down that $$\nabla \pi_\theta(a_t|s_t)=\left(\begin{array}{cccccc} 0.5\times0.5 & -0.5\times0.1 & -0.5\times0.1 & -0.5\times0.1 & -0.5\times0.1 & -0.5\times0.1\\ -0.5\times0.1 & 0.1\times0.9 & -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 \\ -0.5\times0.1 & -0.1\times0.1 & 0.1\times0.9 & -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 \\ -0.5\times0.1 & -0.1\times0.1 & -0.1\times0.1 & 0.1\times0.9 &-0.1\times0.1 & -0.1\times0.1 \\ -0.5\times0.1 & -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 & 0.1\times0.9 & -0.1\times0.1 \\ -0.5\times0.1 & -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 & 0.1\times0.9\\ \end{array}\right)$$ And by multiplying it with $\frac{1}{\pi_\theta(a_t|s_t)}=[\frac{1}{0.5},\frac{1}{0.1},\frac{1}{0.1},\frac{1}{0.1},\frac{1}{0.1},\frac{1}{0.1}]$, you effectively get $$\nabla \log\pi_\theta(a_t|s_t)=\frac{1}{\pi_\theta(a_t|s_t)}\nabla \pi_\theta(a_t|s_t)= \left(\begin{array}{cccccc} 0.5 & -0.5, &-0.5, &-0.5, &-0.5, &-0.5, \\ -0.1 & 0.9 & -0.1 & -0.1 & -0.1 & -0.1 & \\ -0.1 & -0.1 & 0.9 & -0.1 & -0.1 & -0.1 & \\ -0.1 & -0.1 & -0.1 & 0.9 & -0.1 & -0.1 & \\ -0.1 & -0.1 & -0.1 & -0.1 & 0.9 & -0.1 & \\ -0.1 & -0.1 & -0.1 & -0.1 & -0.1 & 0.9 & \\ \end{array}\right)$$

In case you are still thinking that because $0.9>0.5$ in the above result, we are slightly favoring the less likely action (although not 5 times more favorable), I would say that's the desired behavior. Because when we already have a relatively high score for action 1 (pre-activation 0.5) and now increase the score a little bit, we don't want the output probability to increase that much because otherwise the probability would more easily explode out of the range of $[0,1]$.

This kind of saturating gradient effect is very common for anything using the log derivative trick, not only in RL. You can also think of the example in the original REINFORCE algorithm paper (on page 6) where a Bernoulli random variable $$g_i(y_i,p_i)=\left\{ \begin{array}{c l} 1-p_i & \text{if } y_i=0\\ p_i & \text{if } y_i=1 \end{array}\right.$$

has a log derivative as

$$\frac{\partial \log g_i}{\partial p_i}=\left\{ \begin{array}{c l} -\frac{1}{1-p_i} & \text{if } y_i=0\\ \frac{1}{p_i} & \text{if } y_i=1 \end{array}\right.$$ Here when $p_i$ is already large and $y_i=1$, the derivative would be relatively small.

Finally, regarding your question what if we only got such a large large reward because we took the first action? Don't worry, here you are estimating the expected reward (and its gradient) of your current policy $\pi$, not exploiting this policy, so you just want to have a fair (unbiased) estimate, not a best possible reward.

• Thanks! As an intuition for me, will Gaussian policy also over-emphasise, similar to the softmax policy?
– Kari
Sep 15, 2018 at 20:48
• For Gaussian policy, the action space is usually continuous. So a policy is unlikely to ever-emphasis a single action value, but could possibly over-emphasis a small range of action values depending on the choice of variance. Sep 15, 2018 at 21:02
• The $\frac{1}{\pi(s,a)}$ thing is explained nicely here: towardsdatascience.com/… with example code that practically demonstrates why it is important and works Nov 19, 2018 at 13:24