3
$\begingroup$

I am trying to understand the purpose of Dueling DQN. According to this blogpost: enter image description here

our reinforcement learning agent may not need to care about both value and advantage at any given time - this seems to be what I can't understand.

Let's assume we are in state $S_t$ and we select an action which has the highest score. This is the promised total reward that we will get in the future if we take the action.

Notice, we don't yet know the V or A of the future state $S_{t+1}$ (where we will end up after taking this best action), so decoupling V and A in any state, including S_{t+1} seems unnecessary. Additionally, once we do get to work with them we still seem to recombine them during $S_{t+1}$ into a single Q-value, just as was noted in the blog post.

So, to complete my thought: V and A seem to be a "hidden intermediate step", that still gets combined into Q, so we never know it's even there. Even if network somehow benefits from one or the other, how does it help if both streams still end up as Q?

A slightly unrenated thought, 'V' is the score of the current state only. 'A' is the total future expected Advantage, for a particular action, right?

Can someone provide a different example to the sunset?

Edit after answer was accepted:

Found a friendly explanation about this architecture here.

Also, if someone struggles to understand what V, Q and A is, read this answer, and my comment under it.

$\endgroup$
3
$\begingroup$

A slightly unrelated thought, 'V' is the score of the current state only. 'A' is the total future expected Advantage, for a particular action, right?

Not quite. $V$ is the total (discounted) future expected reward, assuming starting in state $s$, following current policy (in a control problem, usually best guess so far at the optimal policy) into the future. That includes selecting current action, $a$, according to the policy being assessed. The advantage function $A(s,a)$ (blog post has the arguments wrong for $A$) is the difference in value between selecting $a$ according to the current policy and selecting a specific action $a$. The value of $A$ also sums all future rewards assuming that the current policy is then followed into the future after this, maybe different, selection. Note that when the policy is the optimal one, and $V(s)$ is accurate, then $A(s,a)$ should always be zero or negative; optimal actions score zero, non-optimal ones will be negative.

Even if network somehow benefits from one or the other, how does it help if both streams still end up as Q? ... Can someone provide a different example to the sunset?

The more prosaic explanation is that the function decomposition is always technically correct (for an MDP). Coding the network like this incorporates known structure of the problem into the network, which otherwise it may have to spend resources on learning. So it's a way of injecting the designer's knowledge of reinforcement learning problems into the architecture of the network.

Conceptually this is similar to designing CNNs for computer vision with local receptive fields because we know edges and textures can be detected this way in images. Although CNNs have more than just that benefit, one of the positive aspects of the design for vision tasks, is that they structurally match known traits of the problem being solved.

Value-based RL control methods (as opposed to policy-gradient methods) work due to "generalised policy iteration", where the agent is constantly assessing the current values of a policy, then using those value estimates in order to make improvements. The split between $V$ and $A$ functions fits very well with that conceptually. The $V$ function is generally being adjusted to assess the current policy as accurately as possible, whilst positive values in $A$ function identify likely changes to the policy.

$\endgroup$
  • $\begingroup$ Thanks for the explanation and I kind of have an intuition of it. May I state that the value function V(S) is more stable and easier to approximate, while due to the complex interactions between S and A, Q(S,A) is much harder to approximate accurately? For example if the agent's car is going to crash into another car it's definitely a bad state, no matter what action will it take next. Hence separately approximating them will at least guarantee that we have a good approximation of V(S)? $\endgroup$ – Shaohua Li Jul 10 '18 at 12:09
  • 1
    $\begingroup$ @ShaohuaLi: I think that is basically correct intuition. There are a few different examples you can construct around that. For instance, if there is an action that the agent could take to avoid the crash, but it does not know it, then V(s) will be low, and all A(s,a) could be close to zero. But if the agent accidentally discovers the action to escape a crash, then first it will generate a high A(s,a) and change the action to take in future. That will eventually lead to V(s) learning that the state is ok, and A(s,a) changing first will encourage the agent to try that action in similar states. $\endgroup$ – Neil Slater Jul 10 '18 at 12:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.