Policy Gradient Methods - ScoreFunction & Log(policy)

Question

In Policy Gradient Methods, Lecture 7 (34:15), David describes a Score Function as being the Gradient of the Log of the policy

Question:

If we have a Neural Network that holds parameters defining our policy, we will have to perform backpropagation to each weight (each parameter). Why would we want to avoid dealing with:

$\nabla_{\theta}\pi_{\theta}(s,a)$

and instead would wish to deal with its re-arranged version:

$\pi_{\theta}(s,a)\nabla_{\theta} log \big( \pi_{\theta}(s,a) \big) $

Don't we still have a $\pi_{\theta}$ that sits inside the log? By the Chain Rule, we will still have to compute $\frac{\partial \pi_{\theta}}{\partial \theta}$ once we arrive to it. Aren't we just postponing inevitable?

---

My Understanding:

I know that in Neural Networks that perform classification, using a Cross-Entropy function (that deals with logs) right after a Softmax Activation would allow for a really neat derivative during backprop: (result - target) Perhaps that's the intent here as well?

Thus, since we are dealing with the Log(policy), am I implied to always use a Softmax function for the last layer of my Neural Net, when working with Policy Gradients? - at 39:37 in the video, David also describes Gaussian Policy. Are these two selected because they work really well with Log, during backprop?

As I understand, in general, the benefits of wrapping a product in a $log()$ is being able to split it into 2 summands: $log(ab) = log(a) + log(b)$ but I can't see where it would be useful in Policy Gradients.

My intuition is this might be useful to prevent vanishing gradients.

Maybe it's related to the fact that it's called "Score Function", therefore we are dealing with a Log for some other reason? In supervised learning we have a Squared Error cost function that provides us a "beginning value", during the backpropagation. Is Score function = cost function?

Answer 1

You have to see this approach more general and not so much from a neural network point of view. The reason one uses the re-arranged version is really just because in the end the gradient of your objective function ends up being an expectation again. This is also pointed out by David Silver at min 44:06.

For an example of the benefits of using $\log$, see UC Berkeley's DeepRL lecture by S. Levine here. The name of the video is "CS294-112 9/6/17"

At some later point, S. Levine also goes into more details regarding the math of the policy gradient and draws comparisons to the maximum likelihood approach with neural networks.