3
$\begingroup$

I am trying to understand the training phase of the tutorial Using Keras and Deep Deterministic Policy Gradient to play TORCS (mirror, code) by Ben Lau published on October 11, 2016.

The tutorial says:

Then the actor policy is updated using the sampled policy gradient: $$\nabla_\theta J = \frac{\partial Q^{\theta}(s,a)}{\partial a}\frac{\partial \mu(s|\theta)}{\partial \theta}$$

which in the code corresponds to: actor.train(states, grads).

In the actor.train() method, I fail to see where $\frac{\partial Q^{\theta}(s,a)}{\partial a}$ gets multiplied by $\frac{\partial \mu(s|\theta)}{\partial \theta}$.

I did read:

self.params_grad = tf.gradients(self.model.output, self.weights, -self.action_gradient)

where self.action_gradient corresponds to $\frac{\partial Q^{\theta}(s,a)}{\partial a}$, and tf.gradients(self.model.output, self.weights) corresponds to $\frac{\partial \mu(s|\theta)}{\partial \theta}$, but I see no multiplication.

Where does $\frac{\partial Q^{\theta}(s,a)}{\partial a}$ get multiplied by $\frac{\partial \mu(s|\theta)}{\partial \theta}$?

$\endgroup$

1 Answer 1

3
$\begingroup$

Look at the documentation of tf.gradients and you will see that the third parameter is used to weight the gradient calculation of first param wrt to second param. This is phrased in the documentation as

A user can provide their own initial grad_ys to compute the derivatives using a different initial gradient for each y (e.g., if one wanted to weight the gradient differently for each value in each y).

So it essentially uses the grad_ys parameter as an element-wise multiplication.

The parameter update gradient calculation in the project code is feeding in the already-calculated value of $\frac{\partial \mu(s|\theta)}{\partial \theta}$ as a placeholder, and using it for this third param.

So the multiplication of the two gradient factors occurs within tf.gradients.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.