Reinforcement learning: understanding this derivation of n-step Tree Backup algorithm

I think I get the main idea, and I almost understand the derivation except for this one line, see picture below:

I understand what we're doing by using the policy probability to weight the rewards from time t + 2 (because getting here depends on the prob of taking an action that gets here). But I don't understand why we similarly subtract the value function from the return...

It also doesn't seem to match the example target return (G) implied for 2 step backup on slide 15 of this lecture's slides:

Thanks for any insight. I could be missing something simple/obvious as I dive into these details.

EDIT - for more context, see pg. 160 of this pdf which is where the picture comes from: http://incompleteideas.net/sutton/book/bookdraft2016sep.pdf

• The link to the book is now broken. New draft can be found at incompleteideas.net/book/the-book-2nd.html but i don't have time at the moment to chase up the diagram. If you or someone else does, please find it and edit the question. Dec 7, 2017 at 16:58

The slides and the book are consistent. Notice how in the slides there is a restriction in the summatory: i.e. $a \neq A_{t+1}$. For $G^{(2)}$, you need to "remove" from $V_{t+1}$ the term that should not be there, i.e. $A_{t+1}$.

Now, why this term is removed?

If you keep this term you will be adding $A_{t+1}$ twice. In 1-step backup, it is part of the expectation of step $S_{t+1}$.

When you calculate 2-step backup you want to replace $(S_{t+1}, A_{t+1})$ in the 1-step expectation with the discounted expected value of $S_{t+2}$. So you substract the term and add the discounted expectation for $S_{t+2}$

• Thanks for your response. Why should that A(t+1) term be removed exactly?
– JDS
Dec 22, 2016 at 18:08
• IS it because we took action A(t+1) so in the backup we subtract the expected reward we could have gotten from the other possible states? Just wanna make sure I have the right interpretation.
– JDS
Dec 22, 2016 at 18:18
• Yes. I have just improved the answer and explained this. Dec 22, 2016 at 18:22