2
$\begingroup$

I've read several times that Policy-based RL methods can work with continious action space, rather than with discrete actions (move left 5 meters, move right 5.5 meters), like Value-based methods (Q-learning)

If Policy-based methods produce probability of taking an action from a current state $S_t$, how can such an action be continuous? We might get 4 probabities for our actions, but we still have to choose a single action from from four:

$A_1: 10\%$

$A_2: 35\%$

$A_3: 5\%$

$A_4: 50\%$

Thus, it's not obvious how can my action be something continuous like: "turn +19.2345 angles clockwise". Such an action must have already been pre-defined to the "19.2345" value, right?

$\endgroup$
4
$\begingroup$

The main requirement of on-policy policy gradient methods is that they use a parametric policy $\pi(a|s, \theta)$ that is differentiable with respect to the parameters $\theta$.

This is not restricted to only describing discrete probability distribution functions (e.g. softmax output layer of neural network). A description of any probability distribution function that is differentiable and possible to sample from is all that is required. This is true for the Normal distribution for instance, so one relatively common solution in continuous spaces is for a neural network to output the mean and standard deviation for the distribution of each component of the action vector that accepts continuous values.

Typically the neural network does not perform the sampling to choose an action. This is also true for a softmax output - it is only additional code, outside of the NN, that interprets the values and selects the action. In addition, and unlike softmax, the NN does not need to directly represent the probability distribution function, just enough data to drive the sampling process. However, the nature of the distribution function does need to be taken into account when calculating the gradient in policy gradient methods.

Thus, it's not obvious how can my action be something continuous like: "turn +19.2345 angles clockwise". Such an action must have already been pre-defined to the "19.2345" value, right?

What the policy might output here, is the two parameters of the distribution $\mathcal{N}(\mu, \sigma)$, which you then must sample to get an action like "turn x degrees clockwise". So for example, the neural network could output $(25, 7)$ and then additional code will interpret those values as describing the distribution and take a sample. If you got a mean 25, standard deviation 7, then at the point you select the action you could get "turn +19.2345 degrees clockwise" amongst a range of other values. The value 19.2345 does not need to be pre-defined or represented in the neural network in order to do that.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks! So a network can literally output raw values, without any activation functions when some precise angle via a "rotation action" is required? In this case, is the $0.5$ in $(19.3, 0.5)$ will be interpreted as the "probability to select this action", and the $19.3$ is the raw value that must be done by such an action? Following third paragraph of your answer: do I understand correctly that 0.5 is the non-activated output of a neuron, because probabilities are cumulative and don't have to add-up to 1? $\endgroup$ – Kari Aug 26 '18 at 11:58
  • 1
    $\begingroup$ @Kari: The $0.5$ in $(19.3, 0.5)$ from the example is the standard deviation for use in a normal distribution. It is not a probability. You would typically use linear output for neurons in this case, so yes it is non-activated. $\endgroup$ – Neil Slater Aug 26 '18 at 11:58
  • 1
    $\begingroup$ @Kari: I have tried to make that clearer by changing the example showing the usual symbols for mean $\mu$ and standard deviation $\sigma$ plus values which are obviously not probabilities nor too close to your example output. $\endgroup$ – Neil Slater Aug 26 '18 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.