I am going through the ddpg baseline code to try and gain an intuitive understanding of how the actor and critic networks function.
DDPG has two components: the actor which is the deterministic policy
\pi and the critic which is the state-value function
Q(s, a). The way you update the actor
\pi is by computing the gradient of
Q(s, \pi(s)). The idea is that the policy can be seen as a continuous equivalent of
argmax and so you try to update it such as it takes the action that maximizes the Q-function in a given state.
This can be depicted as shown below.
The code shows three different neural networks created.
actor_tf = create_neural_net(observations) # Maps states to desired actions
critic_tf = create_neural_net(observations, actions) # Updates value function
critic_with_actor_tf = create_neural_net(observations, actor_tf) # Used for policy updating
My question is with how the policy is updated, and more specifically with
As explained here,
So critic_with_actor_tf represents
Q(s,\pi(s)) the action-state value in a state
observation = state) following the policy
pi (the actor) (
a = \pi(s)). This is what is used to compute the gradient for the actor:
self.actor_loss = -tf.reduce_mean(self.critic_with_actor_tf)
So, it seems like the actor updated by reducing the mean of
This raises the question, what does the TD Error shown in the diagram above represent and how is that related to updating the policy?