# RL's policy gradient (REINFORCE) pipeline clarification

I try to build a policy gradient RL machine, and let's look at the REINFORCE's equation for updating the model parameters by taking a gradient to make the ascent (I apologize if notation is slightly non-conventional):

$$\omega = \omega + \alpha \cdot \nabla_\omega log\ \pi(A_t|S_t) * V_t$$

My questions I am unsure about are the following:

1. Do I calculate the gradient values at each time step $t$ (like in SGD fashion) or averaging gradient over all timesteps of the episode is a better option?
2. Do I care about getting the gradient values of the selected action probability output only (ignoring the outputs for other actions, in a discrete case)? In other words, do I consider the $V_t$ term for non-selected actions to be 0, which make the gradient values equal 0 as well?
3. In a discrete case the cross-entropy (the loss) is defined as: $$H(p,q) = -\sum_x P(x) * log Q(x)$$

Does that mean that if I substitute the labels (denoted as $P(x)$) with the $V_t$ terms (non-zero for selected action only) in my neural network training, I will be getting the correct gradient values of the log-loss which fully satisfy the REINFORCE definition?

For notation and visualizations please take a look at this excellent tutorial Policy Gradients.

1. The second is correct. In PGs we try to maximize the expected reward. In order to do this we approximate the expectation with the mean reward over samples of trajectories under a parametrized policy. In other words sample actions and get their respective rewards over a period of time within an episode. Compute the discounted return from last step back to the first step (this is your $$V_t$$ in your notation which usually you will find it as $$R_t$$ and is the discounted return). Multiply returns with logits and sum. Take a look at slides 8 and 9 of the (1) to see how the REINFORCE is being implemented along with this code examples (lines 59-75).

2. As you may have already realized, no. It's a return over an episode (multiple timesteps) and is calculated as a discounted sum of all the rewards that you collected. Even if you get 1 at the end and 0 everywhere else the reward is propagated back to the first step (doing it by hand helps a lot!) so all rewards at every timestep are converted into returns from that state and timestep.

3. Look at slide 13 of (1). Your intuition is correct. The maximum likelihood equation for a classification problem (cross-entropy) multiplied by the return equates the policy gradient loss. If you use a simple Neural Network with REINFORCE with two actions to perform a task you will notice that the gradients propagated back are the same as in a classification task but here are multiplied with their respective return (line 69 2) instead of the class label (0/1).

A bit of intuition for the 3: This is with accordance with PG methods considered as Model-free methods and map states to actions. If you use a clustering technique to cluster the hidden representations of the last layer of your network (after dimensionality reduction) and color the data points according to which action your network chose, you will find out that the representations are naturally clustered into separate groups depending the action. At a high level, you can state that the REINFORCE performs a type of classification having as label the reward signal.

Hope it helps!

• Thank you. It did help somehow, but do you mind me asking for a little detail? When it comes to (1), " In order to do this we approximate the expectation with the mean reward over samples of trajectories under a parametrized policy.", pls clarify, $\mathbb{E}$ of the total reward of an episode $\tau$ suggests that many $\tau$s are averaged as an estimation of that, however, the vanilla form of gradient deals with just the sum of rewards (or expected rewards) of one $\tau$, and we don't take mean of the rewards over many episodes, right? Sep 21, 2018 at 8:06
• On (2), I think there has been a misunderstanding. I meant to say that for each timestep $t$ of a given episode we only care about taking $log \pi$ corresponding to a chosen action $A_t(S_t)$, which means we do not summarize all the logits for all the actions (not)taken in state $S_t$, is it ok? Sep 21, 2018 at 8:07
• For the (1) you average over timesteps of an episode. If you look at the ref (1) p. 8 you will see that the grad of the expected reward is taken over an episode (or a time horizon). This later is estimated as the mean over the sampled returns multiplied by the logits. Let me know if this is not clear. For (2) your understanding is correct. Sep 21, 2018 at 16:10
• For the (3) i cannot tell you with certainty yes/no as I am not sure how high level or low level is your code and which libraries you are using. In tensorflow for example your actor loss function would be something like this: policy_loss = -tf.reduce_mean(selected_log_probs.total * self.placeholders.advantage). you do not need to do derivations unless you want to implement everything by yourself. (Important: In any case, if you use high level libraries of course do not seek standard supervised learning code and modify it for RL, just build or find one specifically for RL.) Sep 21, 2018 at 16:14
• Thank you very much. As for (3), I build RL almost from scratch, I started with Q-learning: QN, then DQN, then DDQN, using R and mxnet. Lessons learned, and the result seemed worse then expected to me, which is why I will start to build PG step by step to understand better what I do, and better test the solution. Hope it will work better. Sep 22, 2018 at 11:04