Understanding REINFORCE loss

Question

The loss used in REINFORCE algorithm is confusing me.

From Pytorch documentation :

loss = -m.log_prob(action) * reward

We want to minimize this loss.

---

If a take the following example :

• Action #1 give a low reward (-1 for the example)
• Action #2 give a high reward (+1 for the example)

Let's compare the loss of each action considering both have same probability for simplicity :

p(a1) = p(a2)
=> m.log_prob(a1) = m.log_prob(a2)
And loss(a1) = -m.log_prob(a1) * reward(a1) = m.log_prob(a1)
And loss(a2) = -m.log_prob(a2) * reward(a2) = -m.log_prob(a2) = -m.log_prob(a1)
Then loss(a1) < loss(a2) since m.log_prob(X) < 0

Here I don't understand this conclusion : the loss being minimized, it means a small loss is good compared to a high loss.

So here it would mean action #1 is good compared to action #2 ? But the reward says the opposite !

Answer 1

I think the intuition here is that you want to push the negative loglikelihood (NLL) * the reward to be as negative as possible. Since our reward is often not differentiable because it is obtained through sampling, we can only change the NLL. For actions with high reward, we have greater "pressure" / gradient to push the NLL down, than if the reward was low, as the reward acts as a multiplier on the gradient on the NLL. So you'd imagine at convergence/equilibrium, the high reward actions would have greater likelihood than the low reward ones. Let me know if that makes sense!

Answer 2

The actual loss is supposed to be $m.log\_prob(action) * reward$ without the negative sign. The default optimizer in Pytorch uses gradient descent methods, while the REINFORCE assumes gradient ascent update rule. To account for this the loss is made negative. This is clearly mentioned in the documentation. To get the intuitive understanding, you can remove the negative sign.