I think I get the main idea, and I almost understand the derivation except for this one line, see picture below:
I understand what we're doing by using the policy probability to weight the rewards from time t + 2 (because getting here depends on the prob of taking an action that gets here). But I don't understand why we similarly subtract the value function from the return...
It also doesn't seem to match the example target return (G) implied for 2 step backup on slide 15 of this lecture's slides:
Thanks for any insight. I could be missing something simple/obvious as I dive into these details.
EDIT - for more context, see pg. 160 of this pdf which is where the picture comes from: http://incompleteideas.net/sutton/book/bookdraft2016sep.pdf