1
$\begingroup$

In RL Course by David Silver - Lecture 7: Policy Gradient Methods, David explains what an Advantage function is, and how it's the difference between Q(s,a) and the V(s)

enter image description here

Preliminary, from this post:

First recall that a policy $\pi$ is a mapping from each state, $s$, action $a$, to the probability $\pi(a \mid s)$ of taking action $a$ when in state $s$.

The state value function, $V^\pi(s)$, is the expected return when starting in state $s$ and following $\pi$ thereafter.

Similarly, the state-action value function, $Q^\pi(s, a)$, is the expected return of when starting in state $s$, taking action $a$, and following policy $\pi$ thereafter.

In my understanding, $V(s)$ is always larger than $Q(s, a)$, because the function $V$ includes the reward for the current state $s$, unlike $Q$. So, why is the advantage function defined as $A = V - Q$ rather than $A = Q - V$ (at minute 1:12:29 in the video)?

Actually, V might not be larger than Q, because $s$ might actually contain a negative reward. In such a case how can we be certain what to subtract from what, such that our Advantage is always positive?

$Q(s, a)$ returns a value of entire total reward that's expected ultimately, after we pick an action $a$. $V(s)$ is the same, just with an extra reward from current state $s$ as well.

I don't see why a value of $Q - V$ would be useful. On the other hand, $V - Q$ would be useful because it would tell us the reward we would get on $s_{t+1}$ if we took the action $a$.

$\endgroup$
3
$\begingroup$

In my understanding, $V(s)$ is always larger than $Q(s,a)$, because the function $V$ includes the reward for the current state $s$, unlike $Q$

This is incorrect. There is not really such a thing as "the reward for current state" in the general case of a MDP. If you mean the $V(S_t)$ should include the value of $R_t$, then this is still wrong, given David Silver's use of the conventions for time step indexing. It is possible to associate immediate reward with either the current time step, leading to sequence $S_0, A_0, R_0, S_1, A_1, R_1$ etc or you can use the convention of immediate reward being on next time step $S_0, A_0, R_1, S_1, A_1, R_2$ etc. David Silver (and Sutton & Barto's book) uses the latter convention.

Under that convention:

$$V(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1}|S_t=s]$$

$$Q(s,a) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1}|S_t=s, A_t=a]$$

You can see that the first term in the expansion of the sum for both Q(s,a) and V(s) is $R_{t+1}$. If you changed the convention, then both would include the equivalent value, but would be labelled $R_{t}$ in any formula.

Q and V do not differ in which time steps they sum reward over. They may differ in the value of $R_{t+1}$ because $V(s)$ assumes following the policy $\pi$ when selecting $A_t$ whilst $Q(s,a)$ uses the value $a$ supplied as a parameter for $A_t$, which can be different.

how can we be certain what to subtract from what, such that our Advantage is always positive?

Advantage can be negative, that is fine. It means that the action $a$ in $A(s,a)$ is a worse choice than the current policy's.

I don't see why a value of $(Q−V)$ would be useful. On the other hand, $(V−Q)$ would be useful

Both would be equally useful, it is mainly convention that we work with finding maximum Advantage representing the benefit of choosing a specific action instead of following the current policy, as opposed to finding the minimum "Disadvantage". However, the concept of Advantage in this context is arguably the more natural view.

because it would tell us the reward we would get on $s_{t+1}$ if we took the action $a$

As explained above, this is wrong. The value in $A(s,a)$ expresses the potential benefit we would get for changing the policy $\pi(s)$. That might include changes to $R_{t+1}$, but is not limited to a single time step.

Some RL approaches do create a predictive function for expected immediate reward $\hat{r}(s,a)$ - typically this is a secondary component, used to help refine parameters for other function approximators.

$\endgroup$
  • 1
    $\begingroup$ Got it! So my intuition is correct, with one adjustment. As you've explained, V(s) and Q(s,a) include rewards from the next timesteps, onwards. It's just that V(s) promises that we will act according to the policy, even the current timestep, while Q(s,a) allows to select the action a once, right now, and then will follow the policy $\endgroup$ – Kari Sep 2 '18 at 13:35
  • $\begingroup$ Taking such an action $a$ might mean violating the policy once, but allows us to see if such a violation resulted in better total reward (might have placed us on a completely different train of states, with larger rewards). And a difference with $V(s)$ in other words "with total remaining rewards were we to follow that old policy", is exactly the advantage. So $Q - V$ indeed makes sense. Thanks!! $\endgroup$ – Kari Sep 2 '18 at 13:35
  • 1
    $\begingroup$ @Kari: Yes, that's it. $\endgroup$ – Neil Slater Sep 2 '18 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.