Trouble understanding the partial differentiation used in reinforcement learning

Question

I am studying deterministic actor-critic algorithms in reinforcement learning.

I try to give a brief explanation of actor-critic algorithms before jumping into the mathematics. The actor takes in state $s$ and outputs a deterministic action $a$ based on the distribution policy $u$.

The state and action are fed into the critic. The critic sees how good it is to take a particular action from a given state using the action-value function $Q(s,a,w)$.

The critic is then updated via temporal difference (TD) learning and the actor updated in the direction of the critic

Thus it can be seen that the actor's goal is to try and maximise the state action value function $Q(s,a,w)$ by picking the best actions in the given state.

---

I am having trouble understanding the mathematics behind the updating of the actor.

The below equation gives how the actor is updated.

\begin{equation} \frac{\partial l}{\partial u} = \frac{\partial Q(s, a, w)}{\partial a} \frac{\partial a}{\partial u} \end{equation}

What I understand is that we are taking the partial derivative of $l$ with respect to $u$, and we are backpropogating the critic gradient to the actor.

It seems that $l$ is a differentiable function of the variable $a$, but I am confused when it comes to describing what is happening in the equation above as it seems to consist of two functions multiplied together.

Can someone kindly explain what is really happening in the mathematics above?

Answer 1

Your understanding of what's going on seems to be correct, just one little clarification: $u$ should be the model parameters of the deterministic policy $\mu(s,u)$ and not a distribution itself, same as $w$ are the model parameters of $Q(s,a,w)$, but that's probably what you meant (or I might be unfamiliar with the formulation).

Concerning your actual question, the update step implied by $\frac{\partial l}{\partial u}$ is supposed to make the deterministic policy $\mu(s,u)$ to get closer to the optimal $a$, which maximizes $Q(s,a,w)$. As $a = \mu(s,u)$, we have a composite function on our hands

$$ Q(s,a,w) = Q(s, \mu(s, u), w)$$

When updating the actor paramters $u$ such that $Q$ is maximized, we need to make a step in the direction of the gradient of $Q$ with respect to $u$ which, since it is a composite function, is computed using the chain rule

$$ \frac{\partial Q}{\partial u} = \frac{\partial Q}{\partial a}\frac{\partial a}{\partial u}$$

The notation is a bit sloppy, replacing $\mu$ and $a$ and so on, but that also seems to be the case in the literature. So what is going on intuitively consits of two parts:

1. moving $a$ in the direction of $\frac{\partial Q}{\partial a}$ will increase $Q$, e.g. in 1D if $\frac{\partial Q}{\partial a} > 0$ increasing $a$ would increase $Q$ and if $\frac{\partial Q}{\partial a} < 0$ increasing $a$ would decrease $Q$

2. moving $u$ in the direction of $\frac{\partial a}{\partial u}$ will increase $a$, in 1D the example would be the same as above

If you multiply these together and update $u$ according to the product you end up moving $u$ such that $Q$ increases by either increasing or decreasing $a$, which is exactly what you want to do.