Dueling DQN - Advantage Stream, why use average and not the tanh?

Question

For Dueling DQN (page 5), why do authors use an average for Advantage stream, and don't simply "activate" the Advantage stream (with a $tanh$ for example)?

Would "activating" work in theory, and is it a similar idea to what the authors intended to achieve, or am I missing the point?

To remind, this is the equation, which Produces a Q-value for the taken action $a$ by combining the Value stream with the Advantage stream:

$$Q(s,a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \biggl( A(s, a; \theta, \alpha) - \frac{1}{N}\sum_{a'}^{N}A(s, a'; \theta, \alpha) \biggr)$$

where

• $s$ is the current state we are in
• $a$ is the action we've decided to take
• $a'$ is one of any action we could have taken (including the action we've taken)
• $\theta$ are parameters (weights) of the network before the "splitting" into two separate streams
• $\alpha$ are the parameters of Advantage stream
• $\beta$ are the parameters of the Value stream

Answer 1

The goal of these models is to estimate the value of each action choice. They chose to use the average function to estimate the update value because the average function produces a single scalar influenced by each value.

Tanh function is not an appropriate nonlinear activation function. The tanh function only takes a scalar as an input, thus would not weight each value.

The authors tried an appropriate nonlinear activation function for many values - softmax:

We also experimented with a softmax version of equation (8), but found it to deliver similar results to the simpler module of equation (9).