# How to get rid of the expectation in Monte Carlo Policy Gradient method?

Taken from Policy Gradient lecture notes, slide 16 onward:

Here in the David Silver Lecture series I perfectly understand how he took the expectation value in the theorem part by combining the following equation .

But how he removed the expectation part when talking stochastic gradient decent ?

• I don't have time to read the presentation (so I wonder why one equation has r where the other has v_t), but could it be because the SGD estimate of an expectation is simply the value of the individual sample? If you want other people to help you should provide a URL to the PDF.
– Emre
Nov 29, 2017 at 7:26
• www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/pg.pdf From the slide number 16th onward Nov 29, 2017 at 9:27

## 1 Answer

But how he removed the expectation part when talking stochastic gradient decent ?

A result that shows what happens in expectation can be estimated by sampling it, and that is precisely what stochastic gradient descent (or ascent in this case) methods do - they operate on individual samples on the assumption that this will on average produce reasonable direction for optimising parameters.

So there is no need to "get rid of" the expectation. In fact the sequence of equations is working deliberately towards it, because in most situations we do not know the full characteristics of the MDP and cannot calculate $d(s)$ (and maybe not even $\mathcal{R}_{s,a}$) - the expectation is used to remove the need for knowing those terms.

Importantly, $d(s)$ is the probability density of the state = the likelihood on a random sample of all states visited under a certain policy of finding the agent/environment in that state. This is often hard to calculate directly, even if you know the MDP and the policy, and it is intractable if you do not have a model of the MDP. However, when you take many samples then just by the act of sampling and using the value of $s$ that is observed, then you will approximate the true distribution of states in the long run.

• so it is more like in SGA we take one action it's probability and reward in order to calculate the objective function right ? Nov 29, 2017 at 9:34
• You are right it is gradient ascent (SGA) in this case. However, the score function is a direct estimate of the gradient, not of the objective function (the expected return from the policy) - and in fact it is the same gradient for a few different variants of $J$ in this case. Also, no need to estimate probability directly, because the probability already is taken account of when you sample (samples of next state and reward will occur according to their probability, just through the act of sampling). Nov 29, 2017 at 9:44
• Final question : The equation that defines the sum of rewards , has to multiply the reward we get (If this is single step case) and the probability of action given the state . Which means we take again an expected values by recording what kind of actions took from that state . But in the stochastic case it's just we do this action by action right ? so we don't need to get an expected value . Nov 29, 2017 at 10:30
• Correct we don't need to get the expected value or calculate it directly in any way. Just take whatever value you get - the reason the theory shows that the expected value of doing this is the correct one that you want. Just you will have some variance i.e. the individual values could be far from the expected value (actually with basic REINFORCE, often far too much variance, which is why Actor-Critic approach can be better) Nov 29, 2017 at 11:03