RL Advantage function why A = Q-V instead of A=V-Q?

Question

In RL Course by David Silver - Lecture 7: Policy Gradient Methods, David explains what an Advantage function is, and how it's the difference between Q(s,a) and the V(s)

Preliminary, from this post:

First recall that a policy $\pi$ is a mapping from each state, $s$, action $a$, to the probability $\pi(a \mid s)$ of taking action $a$ when in state $s$.

The state value function, $V^\pi(s)$, is the expected return when starting in state $s$ and following $\pi$ thereafter.

Similarly, the state-action value function, $Q^\pi(s, a)$, is the expected return of when starting in state $s$, taking action $a$, and following policy $\pi$ thereafter.

In my understanding, $V(s)$ is always larger than $Q(s, a)$, because the function $V$ includes the reward for the current state $s$, unlike $Q$. So, why is the advantage function defined as $A = V - Q$ rather than $A = Q - V$ (at minute 1:12:29 in the video)?

Actually, V might not be larger than Q, because $s$ might actually contain a negative reward. In such a case how can we be certain what to subtract from what, such that our Advantage is always positive?

$Q(s, a)$ returns a value of entire total reward that's expected ultimately, after we pick an action $a$. $V(s)$ is the same, just with an extra reward from current state $s$ as well.

I don't see why a value of $Q - V$ would be useful. On the other hand, $V - Q$ would be useful because it would tell us the reward we would get on $s_{t+1}$ if we took the action $a$.

Answer 1

In my understanding, $V(s)$ is always larger than $Q(s,a)$, because the function $V$ includes the reward for the current state $s$, unlike $Q$

This is incorrect. There is not really such a thing as "the reward for current state" in the general case of a MDP. If you mean the $V(S_t)$ should include the value of $R_t$, then this is still wrong, given David Silver's use of the conventions for time step indexing. It is possible to associate immediate reward with either the current time step, leading to sequence $S_0, A_0, R_0, S_1, A_1, R_1$ etc or you can use the convention of immediate reward being on next time step $S_0, A_0, R_1, S_1, A_1, R_2$ etc. David Silver (and Sutton & Barto's book) uses the latter convention.

Under that convention:

$$V(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1}|S_t=s]$$

$$Q(s,a) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1}|S_t=s, A_t=a]$$

You can see that the first term in the expansion of the sum for both Q(s,a) and V(s) is $R_{t+1}$. If you changed the convention, then both would include the equivalent value, but would be labelled $R_{t}$ in any formula.

Q and V do not differ in which time steps they sum reward over. They may differ in the value of $R_{t+1}$ because $V(s)$ assumes following the policy $\pi$ when selecting $A_t$ whilst $Q(s,a)$ uses the value $a$ supplied as a parameter for $A_t$, which can be different.

how can we be certain what to subtract from what, such that our Advantage is always positive?

Advantage can be negative, that is fine. It means that the action $a$ in $A(s,a)$ is a worse choice than the current policy's.

I don't see why a value of $(Q−V)$ would be useful. On the other hand, $(V−Q)$ would be useful

Both would be equally useful, it is mainly convention that we work with finding maximum Advantage representing the benefit of choosing a specific action instead of following the current policy, as opposed to finding the minimum "Disadvantage". However, the concept of Advantage in this context is arguably the more natural view.

because it would tell us the reward we would get on $s_{t+1}$ if we took the action $a$

As explained above, this is wrong. The value in $A(s,a)$ expresses the potential benefit we would get for changing the policy $\pi(s)$. That might include changes to $R_{t+1}$, but is not limited to a single time step.

Some RL approaches do create a predictive function for expected immediate reward $\hat{r}(s,a)$ - typically this is a secondary component, used to help refine parameters for other function approximators.

Answer 2

𝑄(𝑠,𝑎) Q ( s , a ) returns a value of entire total reward that's expected ultimately, after we pick an action 𝑎 a . 𝑉(𝑠) V ( s ) is the same, just with an extra reward from current state 𝑠 s as well.

I don't see why a value of 𝑄−𝑉 Q − V would be useful. On the other hand, 𝑉−𝑄 V − Q would be useful because it would tell us the reward we would get on 𝑠𝑡+1 s t+ 1 if we took the action 𝑎 a .

I think OP has a bit of misunderstanding here. Neil has given a very accurate answer, and I will just try to augment it with some math-less explanation, in case any one else has the same query.

We know that every state has a Value attached to it. We further know that every action we take at a state would have a certain value attached to it (which we call the Q value). So we have state-value V(s) and state-action value Q(s,a). Given a policy Π(a|s), the Value of a state can be computed by summing (over all possible actions for that state) the probability of taking an action multiplied with the Q-value of that action. So if a state has just 2 actions with equal probability, then the state value is just the average of the 2 Q-values associated with each action.

𝑄(𝑠,𝑎) Q ( s , a ) returns a value of entire total reward that's expected ultimately, after we pick an action 𝑎 a . 𝑉(𝑠) V ( s ) is the same, just with an extra reward from current state 𝑠 s as well.

I think this is where the confusion on the -ve advantage creeps in. As outlined earlier, V(s) is not Q(s,a) + reward from current state. I had a great deal of confusion w.r.t the Rt+1 nomenclature outlined in Sutton and most other books (though a few have used Rt).

Coming back to OPs question, we now see that V(s) need not be greater than Q(s,a). Maybe it is more for certain actions emanating from that state. Maybe it is less for certain actions emanating from that state. Now let us talk of the advantage function and why that nomenclature is critical. We know that the calculation of state value V(s) takes into account all possible actions and their respective q-values. Say this value is 100. Let us say there are 5 actions emanating from this state. a1 & a4 have q-values in excess of 100 and a2,a3,a5 have q values lesser than 100. So we are at an advantage if we take actions a1 & a4 and the quantum of the advantage is given by the difference between the q-value for that action and V(s). If we want to pick optimal actions, it makes sense to calculate the advantage each action has and pick the ones having advantage > 0. This simply gives the advantage this particular action has over the default policy. Hence Q(s,a)-V(s) is a good definition of the advantage function.