# Understanding the training phase of the tutorial "Using Keras and Deep Deterministic Policy Gradient to play TORCS" tutorial

I am trying to understand the training phase of the tutorial Using Keras and Deep Deterministic Policy Gradient to play TORCS (mirror, code) by Ben Lau published on October 11, 2016.

The tutorial says:

Then the actor policy is updated using the sampled policy gradient: $$\nabla_\theta J = \frac{\partial Q^{\theta}(s,a)}{\partial a}\frac{\partial \mu(s|\theta)}{\partial \theta}$$

which in the code corresponds to: actor.train(states, grads).

In the actor.train() method, I fail to see where $\frac{\partial Q^{\theta}(s,a)}{\partial a}$ gets multiplied by $\frac{\partial \mu(s|\theta)}{\partial \theta}$.

self.params_grad = tf.gradients(self.model.output, self.weights, -self.action_gradient)


where self.action_gradient corresponds to $\frac{\partial Q^{\theta}(s,a)}{\partial a}$, and tf.gradients(self.model.output, self.weights) corresponds to $\frac{\partial \mu(s|\theta)}{\partial \theta}$, but I see no multiplication.

Where does $\frac{\partial Q^{\theta}(s,a)}{\partial a}$ get multiplied by $\frac{\partial \mu(s|\theta)}{\partial \theta}$?

Look at the documentation of tf.gradients and you will see that the third parameter is used to weight the gradient calculation of first param wrt to second param. This is phrased in the documentation as
So it essentially uses the grad_ys parameter as an element-wise multiplication.
The parameter update gradient calculation in the project code is feeding in the already-calculated value of $\frac{\partial \mu(s|\theta)}{\partial \theta}$ as a placeholder, and using it for this third param.
So the multiplication of the two gradient factors occurs within tf.gradients.