The Dueling DQN is a good strategy to avoid your network be overoptimistic. Otherwise it will cause actual DQN and to get as optimistic as it explodes. So the Q-values become larger and larger over time on some games and sometimes they never get back.
https://arxiv.org/pdf/1509.06461.pdf
A couple words about Double Q-Learning
$$Q_1^{\hat{}}(s_t,a_t) = r_t + \gamma*Q_2(s_{t+1}, argmaxQ_1(s_{t+1}, a^*))$$
$$Q_2^{\hat{}}(s_t,a_t) = r_t + \gamma*Q_1(s_{t+1}, argmaxQ_1(s_{t+1}, a^*))$$
If one Q function is not enough to make an accurate approximation, Double Q-learning suggest to learn two networks and train one another. So you implement Q1 and Q2 - two independent estimates of the actual value function. Q1 and Q2 in a DQN case are networks with a different set of weights.
If take an action which is optimal in the first Q function, there won't be any connection between this overoptimism for selected action in Q1, and the same overoptimism in Q2. In fact, the same action in Q2, is more or less independent. It can be overoptimistic or overpessimistic and it can be exactly the true value.
The idea here is that, the noise in Q2 is independent of the noise in Q1. And if you update them that way, then you take the maximization which will take account of the sampling error.
Since they two networks are more or less decorrelated, they have different kinds of noises, then the overoptimism disappears.
The drawback of this method - training two networks separately will double the convergence time.
But a DQN algorithm that can serve as an alternative source of q-values, aside from the main q-network - we take the older snapshot of your network as the source of independent randomness as the other Q network
$$Q^{\hat{}}(s_t,a_t) = r_t + \gamma*Q_{old}(s_{t+1}, argmaxQ(s_{t+1}, a^*))$$
Another huge problem is that another huge problem with DQN is that it actually tries to approximate a set of values that are very interrelated. And double Q-learning can't solve this problem easily.
Dueling DQN
The Dueling DQN can solve the overoptimistic problem as well
$$ Q(s,a) = V(s) - A(s,a)$$
The intuition behind the advantage is how much you're action value differs from the state value.
For example, if you have a state in which you have two actions, the first brings you returns of + 100 and the second is + 1. In this case, both action values are positive because you get positive reward. The advantages on the contrary are different. If you take this suboptimal action, then you get the action value of (+ 1) minus the state value of (+ 100), which is -99. Basically, it tells you that you have just lost 99 potential units of rewards in this case.
To predict advantages, we have to constrain them. In case of $V^*$, the maximum possible advantage value is zero because you can never get action value which is larger than maximum over all possible action values from the state.
So you you train two halves together and then just add them up to get your action values.
could anyone who can help me if we decompose them and combine back them into a single Q, what the network can learn?
from my perspective，the V means the total reward when the agent follow the current policy;the Q means if we give a specific action then follow the current policy what the total reward;
and if we get the optimal policy，the V will equal to Q;so we should learn to make the A reach zero;just like the answer ：Dueling DQN - can't understand its mechanism
