Actor-critic architecture: How is the policy updated?

Question

I am going through the ddpg baseline code to try and gain an intuitive understanding of how the actor and critic networks function.

DDPG has two components: the actor which is the deterministic policy \pi and the critic which is the state-value function Q(s, a). The way you update the actor \pi is by computing the gradient of Q(s, \pi(s)). The idea is that the policy can be seen as a continuous equivalent of argmax and so you try to update it such as it takes the action that maximizes the Q-function in a given state.

This can be depicted as shown below.

The code shows three different neural networks created.

1. actor_tf = create_neural_net(observations) # Maps states to desired actions
2. critic_tf = create_neural_net(observations, actions) # Updates value function
3. critic_with_actor_tf = create_neural_net(observations, actor_tf) # Used for policy updating

My question is with how the policy is updated, and more specifically with critic_with_actor_tf.

As explained here,

So critic_with_actor_tf represents Q(s,\pi(s)) the action-state value in a state s (here observation = state) following the policy pi (the actor) (a = \pi(s)). This is what is used to compute the gradient for the actor:

 self.actor_loss = -tf.reduce_mean(self.critic_with_actor_tf)

So, it seems like the actor updated by reducing the mean of critic_with_actor_tf.

This raises the question, what does the TD Error shown in the diagram above represent and how is that related to updating the policy?

Answer 1

You basically have two problems on your hand:

1. learning the Q-function

2. learning the policy

The Q-functions tells you how rewarding an action $a$ would be in state $s$. Once you take the action $a$ and transition from state $s$ to $s'$, you are rewarded $r$ right away. How rewarding it was to take the action $a$ in $s$ is the sum of the received reward $r$ and the Q-value of the state that you end up in, hence you want to minimize

$$r + Q(s', a', w) - Q(s, a, w)$$

where $w$ are the parameters of the function approximator and $a'$ the action you would take in the following state $s'$. This is the TD Error and mimimizing it will learn the Q-function.

So, the TD Error is used to learn the Q-function, basically the same way as it is done in plain-vanilla (Deep) Q-Learning. But while in plain-vanilla (Deep) Q-learning you would base the actor's policy on the Q-function entirely (plus some randomness to not get stuck), in the actor-critic context the Q-function is just a means to the end of training the policy itself. For this reason, the learned Q-function then serves as an optimization objective for the policy by learning the policy parameters $u$ such that it outputs the action $a$ which maximizes $Q(s,a,w)$ (see your other question). The chain rule pops up in that context, because you are not trying to do regression by minimizing some error, but rather you are trying to maximize $Q$ with respect to one of its arguments.

Answer 2

The TD Error is used to directly update the critic and not the actor.

The actor is updated in the direction of the critic using:

\begin{equation} \frac{\partial l}{\partial u} = \frac{\partial Q(s, a, w)}{\partial a} \frac{\partial a}{\partial u} \end{equation}

Follow up question about what this equation really means here.