How does action get selected in a Policy Gradient Method?

Question

As I understood, in Reinforcement-Learning a big difference between a Value-based method and a Policy-gradient method is how the next action is selected.

In Q-learning (Value-based method), each possible action gets a score. We then select next action that has a highest score, with a slight possibility to select any other action at random.

How does a policy gradient method decide upon an action? We are no longer allowed to select the highest-scored action, as this will leave us with a non-differentiable policy, that can't be improved. The actions still have scores - do we select them at random ...using the "cummulative distribution function"? Also, am I using the right term?

In other words, do actions that have a higher score occupy more space and are more likely to be selected?

Edit: there is a really awesome lecture about policy grads. Unfortunatelly the video is deliberately unlisted, so that normal people can't really get to it - but I am for free education, so here it is: CS294-112 9/6/17

Answer 1

You are on the right track. We no longer select an action that we think maximizes the score. Rather we predict what the best action to take is. This can be very effective in large or continuous state spaces where taking an argmax of the score over all possible actions would be prohibitively expensive.

Our policy network computes the probability of taking an action, and during training we sample actions from the network. This continues until we receive a reward signal, which we multiply by the predicted log probabilities to propagate the error back through the network over past actions leading up to that reward. This will tend to adjust the past actions' predicted probabilities down for more negative rewards, and up for more positive rewards.

Of course not all actions leading up to the reward necessary lead to that good or bad reward, but over many training iterations good actions will tend to lead to more positive reward and negative actions will lead to more negative reward.

For more discussions see Andre Karpathy's blog post Deep Reinforcement Learning: Pong from Pixels and RL Course by David Silver - Lecture 7: Policy Gradient Methods