I want to implement the following algorithm, taken from this book, section 13.6:

enter image description here

Here, the neural networks' outputs are $V(S, w)$ and $\pi(A|S,\theta)$, parameterized by $w$ and $\theta$ respectively.

For this question, I will refer only to $V(S,w)$ for simplicity.

The update rules for w and for theta, the two networks' parameters, depend on Zw and on Ztheta, which in turn depend recursively on themselves, as well as on the networks' outputs' derivatives.

Simplifying the equations to create a loss term [that pytorch can handle] that could be derived using loss.backward() would yield

$w \leftarrow w + \alpha^w \delta z^w_{new}$

$w \leftarrow w + \alpha^w \delta [\gamma \lambda^w z^w_{old} + I \nabla_wV(S,w)]$

$w \leftarrow w + \alpha^w \delta \nabla_w[\gamma \lambda^w z^w_{old} w + I V(S,w)]$

thus the loss would be

$Loss = \delta[\gamma \lambda z^w_{old}w + I V(S,w)]$

A few things I am struggling with:

  1. How to obtain $w$ for the loss term, Meaning, accessing they network's weights directly somehow?
  2. How to write an iterative update rule without the mathematical manipulations? This is prefered for cases when such manipulations are difficult, or impossible, or for cases when the update rule is given in the above form (like in the book), and the hassle of going to the loss form could maybe be avoided. This would also prevent mistakes, by just taking the final form from the (tested) book.
  • $\begingroup$ Just an idea for replacing derivatives: Some direct search approach or genetic algorithms? $\endgroup$ Feb 18, 2019 at 10:25


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy