naive residual-algorithm discribed in book RLAI Chapter 11.5 by Sutton and Barto is worked as:

$$ \begin{aligned} \mathbf{w}_{t+1} &=\mathbf{w}_{t}-\frac{1}{2} \alpha \nabla\left(\rho_{t} \delta_{t}^{2}\right) \\ &=\mathbf{w}_{t}-\alpha \rho_{t} \delta_{t} \nabla \delta_{t} \\ &=\mathbf{w}_{t}+\alpha \rho_{t} \delta_{t}\left(\nabla \hat{v}\left(S_{t}, \mathbf{w}_{t}\right)-\gamma \nabla \hat{v}\left(S_{t+1}, \mathbf{w}_{t}\right)\right) \end{aligned} $$

There are three elements: function approximation, bootstrapping and off-policy training, which called the deadly triad in Chapter 11.3 make the algorithm with the denger of instability and divergence.

But why residual gradient algorithm is stable to converge to a suboptimal? As the per-step update also involves the deadly triad.

I think must something special happen when we minimize Mean Squared TD Error(derive to naive residual-algorithm).

From my point of view, minimize Mean Squared TD Error is an extra constraint to the semi-gradient TD which make naive residual-algorithm stable.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.