0
$\begingroup$

Taken from Policy Gradient lecture notes, slide 16 onward:

enter image description here

Here in the David Silver Lecture series I perfectly understand how he took the expectation value in the theorem part by combining the following equation .

enter image description here

But how he removed the expectation part when talking stochastic gradient decent ?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ I don't have time to read the presentation (so I wonder why one equation has r where the other has v_t), but could it be because the SGD estimate of an expectation is simply the value of the individual sample? If you want other people to help you should provide a URL to the PDF. $\endgroup$
    – Emre
    Commented Nov 29, 2017 at 7:26
  • $\begingroup$ www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/pg.pdf From the slide number 16th onward $\endgroup$
    – Shamane Siriwardhana
    Commented Nov 29, 2017 at 9:27

1 Answer 1

Reset to default
1
$\begingroup$

But how he removed the expectation part when talking stochastic gradient decent ?

A result that shows what happens in expectation can be estimated by sampling it, and that is precisely what stochastic gradient descent (or ascent in this case) methods do - they operate on individual samples on the assumption that this will on average produce reasonable direction for optimising parameters.

So there is no need to "get rid of" the expectation. In fact the sequence of equations is working deliberately towards it, because in most situations we do not know the full characteristics of the MDP and cannot calculate $d(s)$ (and maybe not even $\mathcal{R}_{s,a}$) - the expectation is used to remove the need for knowing those terms.

Importantly, $d(s)$ is the probability density of the state = the likelihood on a random sample of all states visited under a certain policy of finding the agent/environment in that state. This is often hard to calculate directly, even if you know the MDP and the policy, and it is intractable if you do not have a model of the MDP. However, when you take many samples then just by the act of sampling and using the value of $s$ that is observed, then you will approximate the true distribution of states in the long run.

Improve this answer
$\endgroup$
4
  • $\begingroup$ so it is more like in SGA we take one action it's probability and reward in order to calculate the objective function right ? $\endgroup$
    – Shamane Siriwardhana
    Commented Nov 29, 2017 at 9:34
  • $\begingroup$ You are right it is gradient ascent (SGA) in this case. However, the score function is a direct estimate of the gradient, not of the objective function (the expected return from the policy) - and in fact it is the same gradient for a few different variants of $J$ in this case. Also, no need to estimate probability directly, because the probability already is taken account of when you sample (samples of next state and reward will occur according to their probability, just through the act of sampling). $\endgroup$
    – Neil Slater
    Commented Nov 29, 2017 at 9:44
  • $\begingroup$ Final question : The equation that defines the sum of rewards , has to multiply the reward we get (If this is single step case) and the probability of action given the state . Which means we take again an expected values by recording what kind of actions took from that state . But in the stochastic case it's just we do this action by action right ? so we don't need to get an expected value . $\endgroup$
    – Shamane Siriwardhana
    Commented Nov 29, 2017 at 10:30
  • 1
    $\begingroup$ Correct we don't need to get the expected value or calculate it directly in any way. Just take whatever value you get - the reason the theory shows that the expected value of doing this is the correct one that you want. Just you will have some variance i.e. the individual values could be far from the expected value (actually with basic REINFORCE, often far too much variance, which is why Actor-Critic approach can be better) $\endgroup$
    – Neil Slater
    Commented Nov 29, 2017 at 11:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.