# How to get rid of the expectation in Monte Carlo Policy Gradient method?

Taken from Policy Gradient lecture notes, slide 16 onward:

Here in the David Silver Lecture series I perfectly understand how he took the expectation value in the theorem part by combining the following equation .

But how he removed the expectation part when talking stochastic gradient decent ?

• I don't have time to read the presentation (so I wonder why one equation has r where the other has v_t), but could it be because the SGD estimate of an expectation is simply the value of the individual sample? If you want other people to help you should provide a URL to the PDF. – Emre Nov 29 '17 at 7:26
• www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/pg.pdf From the slide number 16th onward – Shamane Siriwardhana Nov 29 '17 at 9:27

So there is no need to "get rid of" the expectation. In fact the sequence of equations is working deliberately towards it, because in most situations we do not know the full characteristics of the MDP and cannot calculate $d(s)$ (and maybe not even $\mathcal{R}_{s,a}$) - the expectation is used to remove the need for knowing those terms.
Importantly, $d(s)$ is the probability density of the state = the likelihood on a random sample of all states visited under a certain policy of finding the agent/environment in that state. This is often hard to calculate directly, even if you know the MDP and the policy, and it is intractable if you do not have a model of the MDP. However, when you take many samples then just by the act of sampling and using the value of $s$ that is observed, then you will approximate the true distribution of states in the long run.
• You are right it is gradient ascent (SGA) in this case. However, the score function is a direct estimate of the gradient, not of the objective function (the expected return from the policy) - and in fact it is the same gradient for a few different variants of $J$ in this case. Also, no need to estimate probability directly, because the probability already is taken account of when you sample (samples of next state and reward will occur according to their probability, just through the act of sampling). – Neil Slater Nov 29 '17 at 9:44