6
$\begingroup$

I have been reading this excellent post: https://medium.com/@jonathan_hui/rl-policy-gradients-explained-9b13b688b146 and following the RL-videos by David Silver, and I did not get this thing:

For $\pi_\theta(\tau) = \pi_\theta(s_1, a_1, ..., s_T, a_T) = p(s_1) \prod_{t=1}^T \pi_\theta(a_t | s_t)p(s_{t+1}|a_t, s_t)$ being the likelihood of a given trajectory in a cycle, the derivative of the value function becomes $$\nabla_{\theta}J = E[\nabla_{\theta}log\pi_{\theta} \cdot r ]$$

which then immediately becomes

$$={{1} \over {N}} \sum_{i=1}^{N}(\sum_t^T \nabla_\theta log \pi_\theta(a_{i,t}, s_{i,t})) r$$

i.e. summed over all N paths for $\tau$, while I expected

$$=\sum_\tau \pi_\theta(\tau) \sum_t^T \nabla_\theta log \pi_\theta(\tau) r$$

What I do not get: Where does the probability for the trajectories $\pi_\theta(\tau)$ (left-most sum) go or why did it get replaced by the mean over all paths? Is it assumed that all trajectories are equally likely, given that you start from a known starting position?

(You can find the equations in the blog-post linked above, at the end of the chapter "Optimization", right before the chapter "Intuition".)

$\endgroup$
2
$\begingroup$

Actually in the article there is a $\approx$ rather than an $=$. This is because you can approximate expectation values by sampling the respective distribution.

Assume you want to compute

$$ E \left[ f(x) \right] = \int p(x) f(x) dx$$

The integral might not be tractable, you might not even know the distribution $p(x)$. But as long as you can sample from $p(x)$, you can approximate quite well using the Monte Carlo estimator

$$ E \left[ f(x) \right] \approx \frac{1}{N} \sum_{i=1}^N f(x_i)$$

with $x_i \sim p(x)$ for all $i$. This approximation would get better for larger $N$. The distribution $p(x)$ is, in some sense, represented by the samples $x_i$ and their respective frequency.

This is what is going on in the article. You want to compute the expectation value over all possible trajectories, but that is infeasible. Luckily though, you can sample the distribution by running simulations of the environment. The expectation is then approximated using samples of trajectories. The $\pi_\theta(\tau)$ is represented in the sampled $a_{i,t}$ and $s_{i,t}$.

In short, the expectation over all possible paths is approximated by the mean over samples of paths.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Oh, that makes sense, thanks a lot! So it indeed is that you assign equal likelihoods to the trajectories, because you sample them and each singular sample is equally likely. Thanks again, this had been tormenting me the past days and I had the fear that I might have misunderstood something very severely. Now it all makes sense. $\endgroup$ – Hadamard Feb 19 at 9:47
  • 1
    $\begingroup$ Glad I could help! I don't think you have to assume the trajectories to be equally likely. They will be represented in the samples according to their likelihood. That's why the simple average over the samples will converge to the true expectation for large N. But I'm not an expert on these details, maybe check out the literature on Monte Carlo estimators and importance sampling for better detail. $\endgroup$ – matthiaw91 Feb 19 at 10:17
  • 1
    $\begingroup$ The trajectories being effectively represented in your collection of samples for a large N is exactly the explanation why you can value each sample with the same weight: Each individual sample is equally likely, but drawing a lot, you will get more trajectories that are similar from a more likely area of the trajectory space. E.g. if you would draw samples from the European population: Each person would have the same likelihood being drawn. However, in the end, you would end up with more dark-haired-white-skinned people than not. $\endgroup$ – Hadamard Feb 19 at 13:26
  • 1
    $\begingroup$ Now I see what you mean. That sounds about right ;) $\endgroup$ – matthiaw91 Feb 19 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.