In value-function approximation, in particular, in deep Q-learning, I understand that we first predict the Q values for each action. However, when there are many actions, this task is not easy.
But in policy iteration we also have to output a softmax vector related to each action. So I don't understand how this can be used to work with continuous action space.
Why are policy gradient methods preferred over value function approximation in continuous action domains?
But in policy iteration also we are have to output a softmax vector related to each actions
This is not strictly true. A softmax vector is one possible way to represent a policy, and works for discrete action spaces. The difference between policy gradient and value function approaches here is in how you use the output. For a value function you would find the maximum output, and choose that (perhaps $\epsilon$-greedily), and it should be an estimate of the value of taking that action. For a policy function, you would use the output as probability to choose each action, and you do not know the value of taking that action.
So I don't understand how this can use to work with continuous action space ?
With policy gradient methods, the policy can be any function of your parameters $\theta$ which:
Outputs a probability distribution
Can be differentiated with respect to $\theta$
So for instance your policy function can be
$$\pi_{\theta}(s) = \mathcal{N}(\mu(s,\theta), \sigma(s,\theta))$$
where $\mu$ and $\sigma$ can be functions you implement with e.g. a neural network. The output of the network is a description of the Normal distribution for the action value $a$ given a state value $s$. The policy requires you to sample from the normal distribution defined by those values (the NN doesn't do that sampling, you typically have to add that in code).
Why are policy gradient methods preferred over value function approximation in continuous action domains?
Whilst it is still possible to estimate the value of a state/action pair in a continuous action space, this does not help you choose an action. Consider how you might implement an $\epsilon$-greedy policy using action value approximation: It would require performing an optimisation over the action space for each and every action choice, in order to find the estimated optimal action. This is possible, but likely to be very slow/inefficient (also there is a risk of finding local maximum).
Working directly with policies that emit probability distributions can avoid this problem, provided those distributions are easy to sample from. Hence you will often see things like policies that control parameters of the Normal distribution or similar, because it is known how to easily sample from those distributions.
In Value function methods (or Critic methods) we usually choose one of the following options to select our actions after we have estimated the relevant value function:
In Policy Gradient (or Actor Methods) we have two approaches:
From the above you can clearly see that when continuous action space is involved, PG offers much more plausible solution.
For more information I suggest you the Master thesis PG Methods: SGD-DPG of Riashat. He worked closely with one of the co-authors of Silver's DPG paper and the thesis is very well structured with MATLAB code available. Policy Gradients methods are much harder to understand because of the math involved (and many times the different notation used). However I would suggest you to start from Jan Peters and then get back to Silver's slides (my opinion :) ).
