In Policy Gradient Methods, Lecture 7 (34:15), David describes a Score Function as being the Gradient of the Log of the policy
Question:
If we have a Neural Network that holds parameters defining our policy, we will have to perform backpropagation to each weight (each parameter). Why would we want to avoid dealing with:
$\nabla_{\theta}\pi_{\theta}(s,a)$
and instead would wish to deal with its re-arranged version:
$\pi_{\theta}(s,a)\nabla_{\theta} log \big( \pi_{\theta}(s,a) \big) $
Don't we still have a $\pi_{\theta}$ that sits inside the log? By the Chain Rule, we will still have to compute $\frac{\partial \pi_{\theta}}{\partial \theta}$ once we arrive to it. Aren't we just postponing inevitable?
My Understanding:
I know that in Neural Networks that perform classification, using a Cross-Entropy function (that deals with logs) right after a Softmax Activation would allow for a really neat derivative during backprop: (result - target)
Perhaps that's the intent here as well?
Thus, since we are dealing with the Log(policy), am I implied to always use a Softmax function for the last layer of my Neural Net, when working with Policy Gradients? - at 39:37 in the video, David also describes Gaussian Policy. Are these two selected because they work really well with Log, during backprop?
As I understand, in general, the benefits of wrapping a product in a $log()$ is being able to split it into 2 summands: $log(ab) = log(a) + log(b)$ but I can't see where it would be useful in Policy Gradients.
My intuition is this might be useful to prevent vanishing gradients.
Maybe it's related to the fact that it's called "Score Function", therefore we are dealing with a Log for some other reason? In supervised learning we have a Squared Error cost function that provides us a "beginning value", during the backpropagation. Is Score function = cost function?