What does this value function of a parameterized policy mean? and is it related to TRPO in RL?

Question

Iv been watching the RL lectures on youtube from Stanford. In episode 9 – policy gradients 2 the teacher Emma Brunskill says we are going to learn about how to make safer policy gradient steps by adjusting (not sure how yet) the policy gradient estimate we compute. The teacher says the equation for the value function of the policy parameterized by $\theta$ is the following.

$$ V(\theta) = E_{\pi\theta} [\sum_{t=0}^{\infty} \gamma^{t} R(s_t,a_t)] $$

where $ s_0 \sim \mu(\pi) $ the start state distribution, $a_t \sim \pi(a_t|s_t) $ the policy, $ s_{t+1} \sim P(s_{t+1}|s_t,a_t)$ the transition model of a MDP that terminates or is episodic, $R(s_t,a_t)$ is the reward model evaluated for the state and action expected at time step t.

Then the teacher says we can write the value function of another policy like the following. I get lost here.

$$ V(\tilde{\theta}) = V(\theta) + E_{\pi\tilde{\theta}}[\sum_{t=0}^{\infty} \gamma^{t} A_{\pi}(s_t,a_t)] $$

then she re-expresses that expectation as.

$$ V(\tilde{\theta}) = V(\theta) + \sum_{s}\mu(s)\sum_{a}\pi(a|s) A_{\pi}(s,a) $$

Where $V(\theta)$ is the value function of the policy parameterized by $\theta$ and $\tilde{\theta}$ is a different set of parameters than $\theta$. I don’t understand why? Im looking for an explanation of what this equation is used for? Or why it makes sense to represent the value function of a policy using the value function and advantage function of another policy.

Does this mean that $V(\tilde{\theta})$ is expected to be different from $V(\theta)$ by the the advantage function of $\pi\theta$ averaged by the distribution of states and actions seen under $\pi\tilde{\theta}$ Im having a hard time wrapping my mind around that. I think it means that if the new policy chooses actions differently than the current policy it will expect to get advantage that the current policy expects to get from taking those actions. (and additionally weighted by the discounted distribution of states seen undere the new policy)

I haven’t read the cource testbook Reinforcement Learning: an introduction by Sutton & Barto. If you have and this was part of the book and you understand why this makes sense please let me know.

Answer 1

The teacher tries to show that we need to update the parameters in a way that the new value estimation under the new parameters is monotonically improving the value and hence the policy. The section that is confusing is when she makes use of the Performance-Difference Lemma Kakade & Langford, 2002. But the proof is omitted and thus the confusion. I think these sources have it in a more clear way along with more detailed proofs:

• Analysis on Policy Difference Equation. This source explains also how from the "time description" you get to the "state visitation distribution" description that confused you.
• p. 20

There are plenty of other sources that have detailed derivation of the specific lemma. If you are interested in RL theory then this paper by Kakade is a review of various such methods such as the Conservative Policy Iteration which is the algorithm that the teacher is trying to get to.