# Policy Gradient Methods - ScoreFunction & Log(policy)

In Policy Gradient Methods, Lecture 7 (34:15), David describes a Score Function as being the Gradient of the Log of the policy

Question:

If we have a Neural Network that holds parameters defining our policy, we will have to perform backpropagation to each weight (each parameter). Why would we want to avoid dealing with:

$\nabla_{\theta}\pi_{\theta}(s,a)$

and instead would wish to deal with its re-arranged version:

$\pi_{\theta}(s,a)\nabla_{\theta} log \big( \pi_{\theta}(s,a) \big)$

Don't we still have a $\pi_{\theta}$ that sits inside the log? By the Chain Rule, we will still have to compute $\frac{\partial \pi_{\theta}}{\partial \theta}$ once we arrive to it. Aren't we just postponing inevitable?

My Understanding:

I know that in Neural Networks that perform classification, using a Cross-Entropy function (that deals with logs) right after a Softmax Activation would allow for a really neat derivative during backprop: (result - target) Perhaps that's the intent here as well?

Thus, since we are dealing with the Log(policy), am I implied to always use a Softmax function for the last layer of my Neural Net, when working with Policy Gradients? - at 39:37 in the video, David also describes Gaussian Policy. Are these two selected because they work really well with Log, during backprop?

As I understand, in general, the benefits of wrapping a product in a $log()$ is being able to split it into 2 summands: $log(ab) = log(a) + log(b)$ but I can't see where it would be useful in Policy Gradients.

My intuition is this might be useful to prevent vanishing gradients.

Maybe it's related to the fact that it's called "Score Function", therefore we are dealing with a Log for some other reason? In supervised learning we have a Squared Error cost function that provides us a "beginning value", during the backpropagation. Is Score function = cost function?

For an example of the benefits of using $\log$, see UC Berkeley's DeepRL lecture by S. Levine here. The name of the video is "CS294-112 9/6/17"
• I think I am starting to understand. So indeed, during differentiation, wrapping $\pi(\tau)$ inside of $log$ allows us to A: detach the "Probability of the next state under the dynamics" and B: detach the "probability of the initial state", from C: the "probability of action given state" all of which sit inside $\pi(\tau)$. We then can focus on the latter, without having to know the values of probabilities A or B. – Kari Sep 10 '18 at 1:09
• In this case it's a little unclear why S. Levine used $\pi (\tau)$ showing that we worked with a trajectory (as I would expect) and David Silver used $\pi (s, a)$ to only show a single step, not a trajectory. Edit actually, I think I now remember David mentioning "this was done for simplification, the trajectories would be a homework". – Kari Sep 10 '18 at 1:43