# Reinforcement Learning: Policy Gradient derivation question

I have been reading this excellent post: https://medium.com/@jonathan_hui/rl-policy-gradients-explained-9b13b688b146 and following the RL-videos by David Silver, and I did not get this thing:

For $$\pi_\theta(\tau) = \pi_\theta(s_1, a_1, ..., s_T, a_T) = p(s_1) \prod_{t=1}^T \pi_\theta(a_t | s_t)p(s_{t+1}|a_t, s_t)$$ being the likelihood of a given trajectory in a cycle, the derivative of the value function becomes $$\nabla_{\theta}J = E[\nabla_{\theta}log\pi_{\theta} \cdot r ]$$

which then immediately becomes

$$={{1} \over {N}} \sum_{i=1}^{N}(\sum_t^T \nabla_\theta log \pi_\theta(a_{i,t}, s_{i,t})) r$$

i.e. summed over all N paths for $$\tau$$, while I expected

$$=\sum_\tau \pi_\theta(\tau) \sum_t^T \nabla_\theta log \pi_\theta(\tau) r$$

What I do not get: Where does the probability for the trajectories $$\pi_\theta(\tau)$$ (left-most sum) go or why did it get replaced by the mean over all paths? Is it assumed that all trajectories are equally likely, given that you start from a known starting position?

(You can find the equations in the blog-post linked above, at the end of the chapter "Optimization", right before the chapter "Intuition".)

Actually in the article there is a $$\approx$$ rather than an $$=$$. This is because you can approximate expectation values by sampling the respective distribution.

Assume you want to compute

$$E \left[ f(x) \right] = \int p(x) f(x) dx$$

The integral might not be tractable, you might not even know the distribution $$p(x)$$. But as long as you can sample from $$p(x)$$, you can approximate quite well using the Monte Carlo estimator

$$E \left[ f(x) \right] \approx \frac{1}{N} \sum_{i=1}^N f(x_i)$$

with $$x_i \sim p(x)$$ for all $$i$$. This approximation would get better for larger $$N$$. The distribution $$p(x)$$ is, in some sense, represented by the samples $$x_i$$ and their respective frequency.

This is what is going on in the article. You want to compute the expectation value over all possible trajectories, but that is infeasible. Luckily though, you can sample the distribution by running simulations of the environment. The expectation is then approximated using samples of trajectories. The $$\pi_\theta(\tau)$$ is represented in the sampled $$a_{i,t}$$ and $$s_{i,t}$$.

In short, the expectation over all possible paths is approximated by the mean over samples of paths.

• Oh, that makes sense, thanks a lot! So it indeed is that you assign equal likelihoods to the trajectories, because you sample them and each singular sample is equally likely. Thanks again, this had been tormenting me the past days and I had the fear that I might have misunderstood something very severely. Now it all makes sense. Feb 19, 2020 at 9:47
• Glad I could help! I don't think you have to assume the trajectories to be equally likely. They will be represented in the samples according to their likelihood. That's why the simple average over the samples will converge to the true expectation for large N. But I'm not an expert on these details, maybe check out the literature on Monte Carlo estimators and importance sampling for better detail. Feb 19, 2020 at 10:17
• The trajectories being effectively represented in your collection of samples for a large N is exactly the explanation why you can value each sample with the same weight: Each individual sample is equally likely, but drawing a lot, you will get more trajectories that are similar from a more likely area of the trajectory space. E.g. if you would draw samples from the European population: Each person would have the same likelihood being drawn. However, in the end, you would end up with more dark-haired-white-skinned people than not. Feb 19, 2020 at 13:26
• Now I see what you mean. That sounds about right ;) Feb 19, 2020 at 13:32