I'm not entirely sure why policy gradients have to be on-policy and have to update using trajectories sampled from the current behaviour. In REINFORCE, the loss function is determined by the log probability of an action times the reward (or discounted reward).
For a state $s$, if I take action $a$ and arrive in state $s'$ I will always see reward $r$. So if I have keep all these values, run an $s$ from the past through my current actor then why can't I update my actor with the new log probability of $a$ and the known reward $r$. I don't need to actually have played and seen a new result.
Can someone correct my understanding?