How do the policy gradient's cost function and gradients work?

Question

I am not a math expert but have a basic understanding of linear algebra, calculus and probability and I understand the math behind back propagation. Currently I am trying to learn about policy gradient algorithm, but I am having difficulty understanding the underlying math. The most commonly used cost functions in the neural network training includes a target and output. For example:

MSE:

$$E_{total} = \sum{\frac{1}{2}(target - output)^2}$$

Log Loss:

$$Error = Output(i) * (1 - Output(i)) * (Target(i) - Output(i))$$

The idea is to find parameters $\theta$ which reduces the distance between the target and the output.

But in policy gradient method the cost function is like this:

$$g = \mathbb E\Big[\sum R_t*\frac{(\partial)} {(\partial\theta)}ln\pi_\theta(a_t|s_t)\Big]$$

What is the target and output in policy gradient's cost function?

How does this cost function get minimized and how does it works?

Answer 1

In policy gradients we are interested in maximizing the expected reward. For this we assume that the expected reward is parametrized by parameters $\theta$ (e.g. a Neural Network. This means that in order to maximize the expected reward we need to find these parameters. In math notation: $$ \theta^{\star}=\arg \max _{\theta} J(\theta) =\arg \max _{\theta} E_{\tau \sim p_{\theta}(\tau)}\left[\sum_{t} r\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)\right] $$

where $\tau$ is a trajectory sampled from policy $p_\theta$. To solve this we need gradient ascent so our parameters are updated: $\theta = \theta + \alpha\nabla J(\theta)$. Thus, if we compute the gradient of the expected reward we will get the proper update of our parameters towards greater expected rewards. You can take a look at Likelihood Ration and REINFORCE which explains analytically the whole optimization process.

To solve this you need the Policy Gradient theorem which will lead you to the equation you have. In other words, by using the gradient form you mentioned we are updating our parameters towards higher expected rewards. Also, PGs are more close to classification (cost-sensitive) rather than regression.

In terms of Neural Networks (and assuming stochastic policies), typically, your inputs are going to be states and your outputs action distribution and/or expected reward, thus mapping states to action probabilities (and/or reward predictions). The additional reward prediction it has been shown that leads to better results than using only the policy loss, as it drives the NN's representations to also predict expected rewards (instead of only action distributions).