# Understanding REINFORCE loss

The loss used in REINFORCE algorithm is confusing me.

loss = -m.log_prob(action) * reward


We want to minimize this loss.

If a take the following example :

• Action #1 give a low reward (-1 for the example)
• Action #2 give a high reward (+1 for the example)

Let's compare the loss of each action considering both have same probability for simplicity :

p(a1) = p(a2)
=> m.log_prob(a1) = m.log_prob(a2)
And loss(a1) = -m.log_prob(a1) * reward(a1) = m.log_prob(a1)
And loss(a2) = -m.log_prob(a2) * reward(a2) = -m.log_prob(a2) = -m.log_prob(a1)
Then loss(a1) < loss(a2) since m.log_prob(X) < 0

Here I don't understand this conclusion : the loss being minimized, it means a small loss is good compared to a high loss.

So here it would mean action #1 is good compared to action #2 ? But the reward says the opposite !

The actual loss is supposed to be $$m.log\_prob(action) * reward$$ without the negative sign. The default optimizer in Pytorch uses gradient descent methods, while the REINFORCE assumes gradient ascent update rule. To account for this the loss is made negative. This is clearly mentioned in the documentation. To get the intuitive understanding, you can remove the negative sign.