1
$\begingroup$

I'm not entirely sure why policy gradients have to be on-policy and have to update using trajectories sampled from the current behaviour. In REINFORCE, the loss function is determined by the log probability of an action times the reward (or discounted reward).

For a state $s$, if I take action $a$ and arrive in state $s'$ I will always see reward $r$. So if I have keep all these values, run an $s$ from the past through my current actor then why can't I update my actor with the new log probability of $a$ and the known reward $r$. I don't need to actually have played and seen a new result.

Can someone correct my understanding?

Thanks

$\endgroup$
2
$\begingroup$

The Policy Gradient theorem states that the gradient of the expected reward is equal to the expectation of the log probability of the current policy multiplied by the reward. Notice that to compute the integral of the expectation we can use Monte Carlo method. For this you would need sampled trajectories. Have you looked at a derivation? The whole theorem comes from mathematical derivations. Check in that page the Assumptions and the Likelihood Ratios (REINFORCE) sections. Hope it helps!

$\endgroup$
1
$\begingroup$

Because we use cumulative reward (or the approximated Q/V of it, but not the immediate reward of (s,a)) in order to calculate gradient-contributed by each trajectory, which we collected by follow the most recent policy.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.