1
$\begingroup$

I was working with continuous system RL and obviously stumbled across this Policy Gradient.

I want to know is this something like cost function for RL? It kinda gives that impression considering we are finding out how efficient the system is as a whole (weighted sum of rewards multiplied by all the policies).

Let's take the example of vanilla PG:

$$g = \mathbb E\Big[\sum R_t*\frac{(\partial)}{(\partial\theta)}ln\pi_\theta(a_t|s_t)\Big]$$

Here, the gradient is nothing but the expected value of the Return (which is nothing but the discounted sum of all the reward) multiplied by the how the policy (the network output) needs to change according to the network weights.

This seems similar to a cost function where we use the total error using cross-entropy (something similar to the information given by return) and then we use this to see how the weights of the neural network can be changed through backpropagation.

Let me know if I've got this right.

$\endgroup$
  • 1
    $\begingroup$ @brian-spiering et al. The author has updated his question. This is a perfectly good question and has an answer, so please put this question off hold. $\endgroup$ – Valentas Sep 19 at 5:06
2
$\begingroup$

Yes there is a cost (score/utility) function and your intuition is correct. In vanilla PG we optimize the expected return $J$ of a trajectory $\tau$ under policy $\pi$ parametrized by $\theta$:

$$\nabla_{\theta} J\left(\tau\right) \approx \frac{1}{N} \sum_{i=1}^{N} \nabla_{\theta} \log \pi_{\theta}\left(\tau_{i}\right) r\left(\tau_{i}\right)$$

(You can find lots of information here: https://spinningup.openai.com/en/latest/algorithms/vpg.html)

The vanilla PG is very closely related to maximum likelihood:

$$\nabla_{\theta} J_{\mathrm{ML}}(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \nabla_{\theta} \log \pi_{\theta}\left(\tau_{i}\right)$$

The two gradients are almost the same except the reward multiplication. This reward is the learning signal. Thinking in terms of backpropagation your gradients are being multiplied by the reward signal. You can think of the vanilla PG as a cost-sensitive classification - with the broad sense.

I will give you a very simple example with Neural Networks in order to demonstrate you the result of the PG learning mechanism. Assume a very simple architecture:

( input --> CNN --> CNN --> fully connected (fc) layer --> out1: V(input), out2: $\pi(a|input)$ )

which can be trained asynchronously or synchronously. The output has two heads one for the expected reward and one for the policy. Let's assume a task in which an agent has to learn to select between two rewarding targets (orange r=10, green r=1). The targets are presented either both of them or one at a time randomly upon completion of an episode.

Assume now that the agent is fully trained. Taking the fc representations from various episodes and running tsne (clustering) we get the following picture:

tsne of fc

This 2D representation tells us that the network has a "clear understanding" when there is only the green target, only the orange target and both of them together. We could colored differently the cluster to get a sense of the expected reward or the action preference (as the fc encodes spatial information, reward information and action preference).

Policy Gradients are mapping states to action distributions (and/or reward predictions). This means that the learned function (the Neural Network) should have the appropriate representations to do that mapping. And this is essentially what is learned by the network: a decision boundary (given input state). Please note that this is a very simplistic example to get an insight of what the network has learned so you can easily draw parallels to the classification case.

$\endgroup$
  • $\begingroup$ It would be good to give one of the forms of $J$ rather than show the gradient of it (which is now in the question). Also worth explaining the difference between $J$ and a typical supervised learning cost function. I am not convinced that I would call $J$ a cost function for this reason, and you may be able to construct other functions for policy gradient methods which look closer to supervised learning cost functions that correct errors related to observations. However, the difference might not be important to OP . . . $\endgroup$ – Neil Slater Sep 19 at 7:22
  • $\begingroup$ I am not sure what do you mean by "one of the forms" of J. Someone can use cost/score/utility/loss function depending on what you optimize. I tend to use loss (as optimizers minimize) or utility function to declare a specific reward function that I want to optimize. I am not sure what do you mean about other functions for policy gradient methods. You can add to the policy loss other functions (e.g. mean squared bellman error) but I believe that the OP is interested in classification as he points at the PG loss which is similar to cross-entropy function used as loss in classification. $\endgroup$ – Constantinos Sep 21 at 3:39
  • $\begingroup$ I mean there a few different versions of $J$ with slightly different formulae. In your answer, you don't at any point show the objective function $J$ or any loss function. You show its gradient (which is derived using the Policy Gradient theorem from definition of $J$). However, the OP has asked to see a loss function. $\endgroup$ – Neil Slater Sep 21 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.