# Reinforcement learning: understanding this derivation of n-step Tree Backup algorithm

I think I get the main idea, and I almost understand the derivation except for this one line, see picture below: I understand what we're doing by using the policy probability to weight the rewards from time t + 2 (because getting here depends on the prob of taking an action that gets here). But I don't understand why we similarly subtract the value function from the return...

It also doesn't seem to match the example target return (G) implied for 2 step backup on slide 15 of this lecture's slides:

Thanks for any insight. I could be missing something simple/obvious as I dive into these details.

EDIT - for more context, see pg. 160 of this pdf which is where the picture comes from: http://incompleteideas.net/sutton/book/bookdraft2016sep.pdf

• The link to the book is now broken. New draft can be found at incompleteideas.net/book/the-book-2nd.html but i don't have time at the moment to chase up the diagram. If you or someone else does, please find it and edit the question. – Neil Slater Dec 7 '17 at 16:58

The slides and the book are consistent. Notice how in the slides there is a restriction in the summatory: i.e. $a \neq A_{t+1}$. For $G^{(2)}$, you need to "remove" from $V_{t+1}$ the term that should not be there, i.e. $A_{t+1}$.
If you keep this term you will be adding $A_{t+1}$ twice. In 1-step backup, it is part of the expectation of step $S_{t+1}$.
When you calculate 2-step backup you want to replace $(S_{t+1}, A_{t+1})$ in the 1-step expectation with the discounted expected value of $S_{t+2}$. So you substract the term and add the discounted expectation for $S_{t+2}$