The loss used in REINFORCE algorithm is confusing me.
From Pytorch documentation :
loss = -m.log_prob(action) * reward
We want to minimize this loss.
If a take the following example :
- Action #1 give a low reward (-1 for the example)
- Action #2 give a high reward (+1 for the example)
Let's compare the loss of each action considering both have same probability for simplicity :
p(a1) = p(a2)
m.log_prob(a1) = m.log_prob(a2)
loss(a1) = -m.log_prob(a1) * reward(a1) = m.log_prob(a1)
loss(a2) = -m.log_prob(a2) * reward(a2) = -m.log_prob(a2) = -m.log_prob(a1)
loss(a1) < loss(a2) since
m.log_prob(X) < 0
Here I don't understand this conclusion : the loss being minimized, it means a small loss is good compared to a high loss.
So here it would mean action #1 is good compared to action #2 ? But the reward says the opposite !