0
$\begingroup$

In Policy Gradient Methods, Lecture 7 (34:15), David describes a Score Function as being the Gradient of the Log of the policy

enter image description here

Question:

If we have a Neural Network that holds parameters defining our policy, we will have to perform backpropagation to each weight (each parameter). Why would we want to avoid dealing with:

$\nabla_{\theta}\pi_{\theta}(s,a)$

and instead would wish to deal with its re-arranged version:

$\pi_{\theta}(s,a)\nabla_{\theta} log \big( \pi_{\theta}(s,a) \big) $

Don't we still have a $\pi_{\theta}$ that sits inside the log? By the Chain Rule, we will still have to compute $\frac{\partial \pi_{\theta}}{\partial \theta}$ once we arrive to it. Aren't we just postponing inevitable?

My Understanding:

I know that in Neural Networks that perform classification, using a Cross-Entropy function (that deals with logs) right after a Softmax Activation would allow for a really neat derivative during backprop: (result - target) Perhaps that's the intent here as well?

Thus, since we are dealing with the Log(policy), am I implied to always use a Softmax function for the last layer of my Neural Net, when working with Policy Gradients? - at 39:37 in the video, David also describes Gaussian Policy. Are these two selected because they work really well with Log, during backprop?

As I understand, in general, the benefits of wrapping a product in a $log()$ is being able to split it into 2 summands: $log(ab) = log(a) + log(b)$ but I can't see where it would be useful in Policy Gradients.

My intuition is this might be useful to prevent vanishing gradients.

Maybe it's related to the fact that it's called "Score Function", therefore we are dealing with a Log for some other reason? In supervised learning we have a Squared Error cost function that provides us a "beginning value", during the backpropagation. Is Score function = cost function?

||improve this question
$\endgroup$
2
$\begingroup$

You have to see this approach more general and not so much from a neural network point of view. The reason one uses the re-arranged version is really just because in the end the gradient of your objective function ends up being an expectation again. This is also pointed out by David Silver at min 44:06.

For an example of the benefits of using $\log$, see UC Berkeley's DeepRL lecture by S. Levine here. The name of the video is "CS294-112 9/6/17"

At some later point, S. Levine also goes into more details regarding the math of the policy gradient and draws comparisons to the maximum likelihood approach with neural networks.

||improve this answer
$\endgroup$
  • $\begingroup$ I think I am starting to understand. So indeed, during differentiation, wrapping $\pi(\tau)$ inside of $log$ allows us to A: detach the "Probability of the next state under the dynamics" and B: detach the "probability of the initial state", from C: the "probability of action given state" all of which sit inside $\pi(\tau)$. We then can focus on the latter, without having to know the values of probabilities A or B. $\endgroup$ – Kari Sep 10 '18 at 1:09
  • $\begingroup$ without this trick, we would need to know values for A and B because they would stay as coefficients due to the chain rule. That would mean we would need to know precise distributions which would be incredibly difficult to estimate (and the A would have to be known for every state) $\endgroup$ – Kari Sep 10 '18 at 1:10
  • $\begingroup$ In this case it's a little unclear why S. Levine used $\pi (\tau)$ showing that we worked with a trajectory (as I would expect) and David Silver used $\pi (s, a)$ to only show a single step, not a trajectory. Edit actually, I think I now remember David mentioning "this was done for simplification, the trajectories would be a homework". $\endgroup$ – Kari Sep 10 '18 at 1:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.