3
$\begingroup$

The loss used in REINFORCE algorithm is confusing me.

From Pytorch documentation :

loss = -m.log_prob(action) * reward

We want to minimize this loss.


If a take the following example :

  • Action #1 give a low reward (-1 for the example)
  • Action #2 give a high reward (+1 for the example)

Let's compare the loss of each action considering both have same probability for simplicity :

p(a1) = p(a2)
=> m.log_prob(a1) = m.log_prob(a2)
And loss(a1) = -m.log_prob(a1) * reward(a1) = m.log_prob(a1)
And loss(a2) = -m.log_prob(a2) * reward(a2) = -m.log_prob(a2) = -m.log_prob(a1)
Then loss(a1) < loss(a2) since m.log_prob(X) < 0

Here I don't understand this conclusion : the loss being minimized, it means a small loss is good compared to a high loss.

So here it would mean action #1 is good compared to action #2 ? But the reward says the opposite !

$\endgroup$

2 Answers 2

2
$\begingroup$

I think the intuition here is that you want to push the negative loglikelihood (NLL) * the reward to be as negative as possible. Since our reward is often not differentiable because it is obtained through sampling, we can only change the NLL. For actions with high reward, we have greater "pressure" / gradient to push the NLL down, than if the reward was low, as the reward acts as a multiplier on the gradient on the NLL. So you'd imagine at convergence/equilibrium, the high reward actions would have greater likelihood than the low reward ones. Let me know if that makes sense!

$\endgroup$
1
  • $\begingroup$ Oh that makes sense ! Thanks for the kind explanation ! $\endgroup$
    – Astariul
    Aug 18, 2020 at 23:33
1
$\begingroup$

The actual loss is supposed to be $m.log\_prob(action) * reward$ without the negative sign. The default optimizer in Pytorch uses gradient descent methods, while the REINFORCE assumes gradient ascent update rule. To account for this the loss is made negative. This is clearly mentioned in the documentation. To get the intuitive understanding, you can remove the negative sign.

$\endgroup$
1
  • $\begingroup$ Thanks for your answer. However I already understood that (as mentioned in my question, we want to minimize the loss). My problem is : if you follow the example I gave, the loss of the good action is higher than the loss for the bad action. So, by minimizing the loss we actually tend to the bad action ? $\endgroup$
    – Astariul
    Aug 8, 2019 at 23:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.