Had a struggle with this recently as well, and here is what I came away with:
I believe you can think of the policy update result as similar to total loss over an episode. Recall that in a vanilla neural net, eg a perceptron, loss will be positive as well, and the optimizer is calculating how much each parameter in theta would have to change to achieve that loss (via gradient descent), and then this is backpropagated or the final update.
The difference here is that in this case we are trying to maximize a reward. Why then does the reward not go to infinity at some point? because there is a finite number of episodes in each simulation, and each action can give a maximum of 1, so there is an upper bound on the total reward given by your setup.
I am not really great on theory, but code snippets make more sense to me, so seeing how they are implemented gives some intuition:
Pytorch version - see the finish_episode() function - which is fairly clear.
Tensorflow Example (sorry I do not know a better one) - see the bottom of the session loop.
Here a simulation is run over several episodes which entail:
For each step in an episode, an action is chosen by sampling policy action scores, and these are stored, along with reward from the game.
At end of episode, a total policy loss is calculated with the policy action log probs and actual rewards (averaging over all steps in this case). Gradients are calculated w.r.t to the final policy loss, and backpropagated.
For reference, here is the [http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf ](original paper) on REINFORCE, see pages 4-5 which explains that the update vector is not necessarily increasing, but "lies in a direction for which [the] performance measure is increasing".