I think I get the main idea, and I almost understand the derivation except for this one line, see picture below:

enter image description here

I understand what we're doing by using the policy probability to weight the rewards from time t + 2 (because getting here depends on the prob of taking an action that gets here). But I don't understand why we similarly subtract the value function from the return...

It also doesn't seem to match the example target return (G) implied for 2 step backup on slide 15 of this lecture's slides:


Thanks for any insight. I could be missing something simple/obvious as I dive into these details.

EDIT - for more context, see pg. 160 of this pdf which is where the picture comes from: http://incompleteideas.net/sutton/book/bookdraft2016sep.pdf

  • $\begingroup$ The link to the book is now broken. New draft can be found at incompleteideas.net/book/the-book-2nd.html but i don't have time at the moment to chase up the diagram. If you or someone else does, please find it and edit the question. $\endgroup$ – Neil Slater Dec 7 '17 at 16:58

The slides and the book are consistent. Notice how in the slides there is a restriction in the summatory: i.e. $a \neq A_{t+1}$. For $G^{(2)}$, you need to "remove" from $V_{t+1}$ the term that should not be there, i.e. $A_{t+1}$.

Now, why this term is removed?

If you keep this term you will be adding $A_{t+1}$ twice. In 1-step backup, it is part of the expectation of step $S_{t+1}$.

When you calculate 2-step backup you want to replace $(S_{t+1}, A_{t+1})$ in the 1-step expectation with the discounted expected value of $S_{t+2}$. So you substract the term and add the discounted expectation for $S_{t+2}$

  • $\begingroup$ Thanks for your response. Why should that A(t+1) term be removed exactly? $\endgroup$ – JDS Dec 22 '16 at 18:08
  • $\begingroup$ IS it because we took action A(t+1) so in the backup we subtract the expected reward we could have gotten from the other possible states? Just wanna make sure I have the right interpretation. $\endgroup$ – JDS Dec 22 '16 at 18:18
  • 1
    $\begingroup$ Yes. I have just improved the answer and explained this. $\endgroup$ – Juan Leni Dec 22 '16 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.