7
$\begingroup$

In value-function approximation, in particular, in deep Q-learning, I understand that we first predict the Q values for each action. However, when there are many actions, this task is not easy.

But in policy iteration we also have to output a softmax vector related to each action. So I don't understand how this can be used to work with continuous action space.

Why are policy gradient methods preferred over value function approximation in continuous action domains?

$\endgroup$
8
$\begingroup$

But in policy iteration also we are have to output a softmax vector related to each actions

This is not strictly true. A softmax vector is one possible way to represent a policy, and works for discrete action spaces. The difference between policy gradient and value function approaches here is in how you use the output. For a value function you would find the maximum output, and choose that (perhaps $\epsilon$-greedily), and it should be an estimate of the value of taking that action. For a policy function, you would use the output as probability to choose each action, and you do not know the value of taking that action.

So I don't understand how this can use to work with continuous action space ?

With policy gradient methods, the policy can be any function of your parameters $\theta$ which:

  • Outputs a probability distribution

  • Can be differentiated with respect to $\theta$

So for instance your policy function can be

$$\pi_{\theta}(s) = \mathcal{N}(\mu(s,\theta), \sigma(s,\theta))$$

where $\mu$ and $\sigma$ can be functions you implement with e.g. a neural network. The output of the network is a description of the Normal distribution for the action value $a$ given a state value $s$. The policy requires you to sample from the normal distribution defined by those values (the NN doesn't do that sampling, you typically have to add that in code).

Why are policy gradient methods preferred over value function approximation in continuous action domains?

Whilst it is still possible to estimate the value of a state/action pair in a continuous action space, this does not help you choose an action. Consider how you might implement an $\epsilon$-greedy policy using action value approximation: It would require performing an optimisation over the action space for each and every action choice, in order to find the estimated optimal action. This is possible, but likely to be very slow/inefficient (also there is a risk of finding local maximum).

Working directly with policies that emit probability distributions can avoid this problem, provided those distributions are easy to sample from. Hence you will often see things like policies that control parameters of the Normal distribution or similar, because it is known how to easily sample from those distributions.

| improve this answer | |
$\endgroup$
  • $\begingroup$ oh ! So in the run time our policy score function will give probability of doing each action then select a action with maximum prob |THIS IS WRONG RIGHT !| Instead of that what it will do is it will create a distribution . Something like normalized distribution . So like in epsilon-greedy policy then the system will do actions according to the distribution . Am I right ? $\endgroup$ – Shamane Siriwardhana Nov 29 '17 at 13:44
  • 1
    $\begingroup$ @ShamaneSiriwardhana: Yes your comment seems right to me. The point is neither action-value function nor policy function directly choose the action, you need to add that as part of the learning algorithm. For action-value you must look at all possible actions and find maximum reward to identify a greedy action (you may just pick randomly sometimes $p(\text{non-greedy})= \epsilon$). For policy function, it outputs a distribution and you always just pick randomly from that distribution. $\endgroup$ – Neil Slater Nov 29 '17 at 13:51
  • $\begingroup$ can you answer to this please datascience.stackexchange.com/questions/25259/… $\endgroup$ – Shamane Siriwardhana Dec 1 '17 at 6:55
  • 1
    $\begingroup$ @ShamaneSiriwardhana: No sorry I cannot answer that one, I don't know it well enough. I do monitor all the questions on the site (it is not too busy. maybe 20 questions a day), and I look for questions that I can answer, so there is no need to link them for me. I hope you find your answer. BTW it may help if you stop using screen shots - instead link the slides as the source and put the text from the slide into the question. It might take some time to get the equations right, but it makes the question much better (and more likely to get up-voted) $\endgroup$ – Neil Slater Dec 1 '17 at 8:25
3
$\begingroup$

In Value function methods (or Critic methods) we usually choose one of the following options to select our actions after we have estimated the relevant value function:

  • Boltzman function with a (inverse) temperature parameter that controls how much we 'listen' the maximum of the value function to select our actions.
  • $\epsilon$-greedy: selects the action with the maximum value function with probability (1-$\epsilon$) and a random one with probability $\epsilon$.

In Policy Gradient (or Actor Methods) we have two approaches:

  1. Stochastic Policy Gradients (SPG): Output is a probability over actions. For your algorithm, the output would be the parameters of a pre-specified distribution (usually Gausian) as Neil described. Then you sample that distribution in a similar way you sample the Boltzman distribution.
  2. Deterministic Policy Gradients (DPG): Output is the value of an action (e.g. speed, height etc).

From the above you can clearly see that when continuous action space is involved, PG offers much more plausible solution.

For more information I suggest you the Master thesis PG Methods: SGD-DPG of Riashat. He worked closely with one of the co-authors of Silver's DPG paper and the thesis is very well structured with MATLAB code available. Policy Gradients methods are much harder to understand because of the math involved (and many times the different notation used). However I would suggest you to start from Jan Peters and then get back to Silver's slides (my opinion :) ).

| improve this answer | |
$\endgroup$
  • $\begingroup$ how do we normally sample from these distributions ? Can you give an example. $\endgroup$ – Shamane Siriwardhana Nov 30 '17 at 7:53
  • $\begingroup$ Oh it's like how we select a better value from the distribution with some set of margins like this right ? google.com.sg/…: $\endgroup$ – Shamane Siriwardhana Nov 30 '17 at 7:55
  • $\begingroup$ I am not sure how many details you need on that. You sample these distributions with the usual sampling methods. For the Boltzmann (discrete action space case) you would need to calculate the cumulative distribution (In MATLAB histc). Then you sample a random number between 0 and 1 and you see where in this histogram this random number falls and you select the action that corresponds to the probability which is closest to your random number. If you are unfamiliar with this let me know please. $\endgroup$ – Constantinos Nov 30 '17 at 19:59
  • 1
    $\begingroup$ For the Gaussian case most programming languages have a one-line command to sample from a Gaussian. For example in MATLAB is mvnrand(mu,sigma). The Gaussian parameters are coming from the output of your policy gradient algorithm (stochastic version of course). Notice that with PG methods you get the value of the action e.g. speed=10km which is suitable for continuous action domains. In the discrete case and value methods described above you get and index which indicates the action (e.g. action1='North', output example from sampling: 1 which corresponds to action North). Hope it helps :) $\endgroup$ – Constantinos Nov 30 '17 at 20:05
  • 2
    $\begingroup$ Correct! In SPG cases the policy network attempts to approximate the probability distribution over actions given state. So you always have a sampling step after the net outputs the probability. From that step you get the action that is returned to the environment. In DPG cases the output of the network is the actual action so no need for sampling. $\endgroup$ – Constantinos Dec 4 '17 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.