Why is DDPG a Policy Gradient Method even though it's actor does not output probability?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
This is explained in the original article where deterministic policy gradient theorem was first proposed, in section 3.3:
The deterministic policy gradient theorem does not at first glance look like the stochastic version (Equation 2). However, we now show that, for a wide class of stochastic policies, including many bump functions, the deterministic policy gradient is indeed a special (limiting) case of the stochastic policy gradient. We parametrise stochastic policies $\Pi_{\mu_\Theta,\sigma}$ by a deterministic policy $\mu_\Theta : S \rightarrow A$ and a variance parameter $\sigma$, such that for $\sigma = 0$ the stochastic policy is equivalent to the deterministic policy, $\Pi_{\mu_\Theta,0} \equiv \mu_\Theta$. Then we show that as $\sigma \rightarrow 0$ the stochastic policy gradient converges to the deterministic gradient (see Appendix C for proof and technical conditions).
Therefore, the Deterministic Policy Gradient method is equivalent to a normal (stochastic) Policy Gradient Method in the limit.
