For Dueling DQN (page 5), why do authors use an average for Advantage stream, and don't simply "activate" the Advantage stream (with a $tanh$ for example)?
Would "activating" work in theory, and is it a similar idea to what the authors intended to achieve, or am I missing the point?
To remind, this is the equation, which Produces a Q-value for the taken action $a$ by combining the Value stream with the Advantage stream:
$$Q(s,a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \biggl( A(s, a; \theta, \alpha) - \frac{1}{N}\sum_{a'}^{N}A(s, a'; \theta, \alpha) \biggr)$$
where
- $s$ is the current state we are in
- $a$ is the action we've decided to take
- $a'$ is one of any action we could have taken (including the action we've taken)
- $\theta$ are parameters (weights) of the network before the "splitting" into two separate streams
- $\alpha$ are the parameters of Advantage stream
- $\beta$ are the parameters of the Value stream