Policy gradient vs cost function

Question

I was working with continuous system RL and obviously stumbled across this Policy Gradient.

I want to know is this something like cost function for RL? It kinda gives that impression considering we are finding out how efficient the system is as a whole (weighted sum of rewards multiplied by all the policies).

Let's take the example of vanilla PG:

$$g = \mathbb E\Big[\sum R_t*\frac{(\partial)}{(\partial\theta)}ln\pi_\theta(a_t|s_t)\Big]$$

Here, the gradient is nothing but the expected value of the Return (which is nothing but the discounted sum of all the reward) multiplied by the how the policy (the network output) needs to change according to the network weights.

This seems similar to a cost function where we use the total error using cross-entropy (something similar to the information given by return) and then we use this to see how the weights of the neural network can be changed through backpropagation.

Let me know if I've got this right.

Answer 1

Yes there is a cost (score/utility) function and your intuition is correct. In vanilla PG we optimize the expected return $J$ of a trajectory $\tau$ under policy $\pi$ parametrized by $\theta$:

$$\nabla_{\theta} J\left(\tau\right) \approx \frac{1}{N} \sum_{i=1}^{N} \nabla_{\theta} \log \pi_{\theta}\left(\tau_{i}\right) r\left(\tau_{i}\right)$$

(You can find lots of information here: https://spinningup.openai.com/en/latest/algorithms/vpg.html)

The vanilla PG is very closely related to maximum likelihood:

$$\nabla_{\theta} J_{\mathrm{ML}}(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \nabla_{\theta} \log \pi_{\theta}\left(\tau_{i}\right)$$

The two gradients are almost the same except the reward multiplication. This reward is the learning signal. Thinking in terms of backpropagation your gradients are being multiplied by the reward signal. You can think of the vanilla PG as a cost-sensitive classification - with the broad sense.

I will give you a very simple example with Neural Networks in order to demonstrate you the result of the PG learning mechanism. Assume a very simple architecture:

( input --> CNN --> CNN --> fully connected (fc) layer --> out1: V(input), out2: $\pi(a|input)$ )

which can be trained asynchronously or synchronously. The output has two heads one for the expected reward and one for the policy. Let's assume a task in which an agent has to learn to select between two rewarding targets (orange r=10, green r=1). The targets are presented either both of them or one at a time randomly upon completion of an episode.

Assume now that the agent is fully trained. Taking the fc representations from various episodes and running tsne (clustering) we get the following picture:

This 2D representation tells us that the network has a "clear understanding" when there is only the green target, only the orange target and both of them together. We could colored differently the cluster to get a sense of the expected reward or the action preference (as the fc encodes spatial information, reward information and action preference).

Policy Gradients are mapping states to action distributions (and/or reward predictions). This means that the learned function (the Neural Network) should have the appropriate representations to do that mapping. And this is essentially what is learned by the network: a decision boundary (given input state). Please note that this is a very simplistic example to get an insight of what the network has learned so you can easily draw parallels to the classification case.