Timeline for How to get rid of the expectation in Monte Carlo Policy Gradient method?
Current License: CC BY-SA 3.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 29, 2017 at 11:42 | vote | accept | Shamane Siriwardhana | ||
| Nov 29, 2017 at 11:03 | comment | added | Neil Slater | Correct we don't need to get the expected value or calculate it directly in any way. Just take whatever value you get - the reason the theory shows that the expected value of doing this is the correct one that you want. Just you will have some variance i.e. the individual values could be far from the expected value (actually with basic REINFORCE, often far too much variance, which is why Actor-Critic approach can be better) | |
| Nov 29, 2017 at 10:30 | comment | added | Shamane Siriwardhana | Final question : The equation that defines the sum of rewards , has to multiply the reward we get (If this is single step case) and the probability of action given the state . Which means we take again an expected values by recording what kind of actions took from that state . But in the stochastic case it's just we do this action by action right ? so we don't need to get an expected value . | |
| Nov 29, 2017 at 9:49 | history | edited | Neil Slater | CC BY-SA 3.0 |
added 549 characters in body
|
| Nov 29, 2017 at 9:44 | comment | added | Neil Slater | You are right it is gradient ascent (SGA) in this case. However, the score function is a direct estimate of the gradient, not of the objective function (the expected return from the policy) - and in fact it is the same gradient for a few different variants of $J$ in this case. Also, no need to estimate probability directly, because the probability already is taken account of when you sample (samples of next state and reward will occur according to their probability, just through the act of sampling). | |
| Nov 29, 2017 at 9:34 | comment | added | Shamane Siriwardhana | so it is more like in SGA we take one action it's probability and reward in order to calculate the objective function right ? | |
| Nov 29, 2017 at 8:21 | history | answered | Neil Slater | CC BY-SA 3.0 |