1
$\begingroup$

Could someone explain why the target of the DDPG's policy is $Q(s,\mu(s))$?

My understanding of DDPG is like this. Since it is intractable to calculate the $argmax_a Q(s,a)$ in a continuous space, DDPG uses an universal function estimator (Neural Network) to learn and predict the best action that realize $maxQ(s,a)$ output.

So, my question is that what is the actual target when DDPG trains $\mu(s)$? I thought that it should be an actual action that gives the highest Q-value at given state $s$ ($argmax_aQ(s,a)$). However, in OpenAI spinning up, it says it can make approximation that $max_aQ(s,a)\approx Q(s,\mu(s))$, and says the loss of the policy is $E[Q(s, \mu(s))]$.

What does this mean? Are they targeting the Q-value, instead of action-value? How can this policy can learn the best action?

$\endgroup$
1
$\begingroup$

In DDPG, the actor network is used to calculate the action that maximises the expected reward in a given state, i.e. the action that maximises the $Q$ function. Critic network is used to calculate the action value in a given state. So the critic network is updated just like the way in DQN. While updating the actor however, we don't have a supervised target action (actual action that maximises the Q function) and our aim is to generate the action that maximises $Q(s,a)$, So we generate an action using actor, apply it to the critic function, then modify the actor params to maximise the $Q(s,a)$, i.e. the gradients flow all the way back from Q network to the actor and the update is done using gradient ascent. So, the policy network is not trained independently, it is done with the help of critic network using the $Q$ value as the target and I'd say the loss of the policy is $-E[Q(s, \mu(s))]$.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.