# Policy Gradient with continuous action space

How to apply reinforce/policy-gradient algorithms for continuous action space. I have learnt that one of the advantages of policy gradients is , it is applicable for continuous action space. One way I can think of is discretizing the action space same as the way we do it for dqn. Should we follow the same method for policy -gradient algorithms also ? Or is there any other way this is done?

Thanks

• I'd recommend checking out this related question/answer: datascience.stackexchange.com/a/25212/75152 That answer contains a good description of how policy gradient can be applied to a continuous action space. – zachdj Oct 14 '19 at 13:10
• Hi, thanks for the link, i had a little difficulty understanding some parts, so what I want to confirm is, even in policy gradients for continuous actions we discretize them into discrete actions. Is my understanding correct? – cvg Oct 14 '19 at 14:06
• No that's not correct - there's no need to discretize the action space with policy gradient. For discrete problems, the policy returns a vector of probabilities for each action. But in continuous problems, the policy returns a continuous distribution over the action space. – zachdj Oct 14 '19 at 14:31
• okay, thanks!! @zachdj – cvg Oct 14 '19 at 15:18

We assume that the action distribution is guassian, i.e. that we need to learn the parameters $$\theta$$ of $$\mathcal{N}(a|\mu_\theta,\sigma_\theta)$$. Let's say that $$\theta$$ is given by the weights of a neural network, which we find by optimizing the objective $$\max_\theta \mathbb{E}_{p_{\theta}}\left[ R(s,a)p_\theta(a|s)\right],$$ where $$p_\theta(s,a) = \mathcal{N}(a|\mu_\theta, \sigma_\theta)$$ and $$R(s,a)$$ is the cumulative discounted reward. The gradient is then per policy gradient theorem simply $$\mathbb{E}_{p_{\theta}}\left[\nabla_\theta R(s,a) \log p_\theta(a|s) \right]$$.
In practice we design a neural network to output one $$\mu$$ per action dimension and $$\sigma$$ can either be learned or kept fixed. If learned, interpret the output as $$\log \sigma$$, so it can take en value. To sample the action we use this outputs learned by our network.