Dueling DQN what does a' mean?

Question

what does $a'$ mean in the "combining" equation in Dueling DQN? (top of the page 5)

$$Q(s,a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \biggl( A(s, a; \theta, \alpha) - \frac{1}{N}\sum_{a'}^{N}A(s, a'; \theta, \alpha) \biggr)$$

Where there are $N$ actions to choose from;

• $s$ is the incoming state (the input vector)
• $a$ is the action taken? (the chosen action)
• $a'$ I don't know what it represents in this context
• $\theta$ represents the weights of the convolutional layers
• $\alpha$ are the weights of the "Advantage stream" which outputs a vector
• $\beta$ are the weights of the Value stream (which outputs a scalar)

Why not to simply use $a$ everywhere, why is $a'$ used in the average?

Answer 1

It is just a type of namespacing, because $a$ is already assigned the chosen action. There are two contexts of action being considered in the equation, so there needs to be a symbol for each context. Using $a'$ is an obvious choice as the letter $a$ is implicitly linked to representing an action already.

The sum over $a'$ is a sum over all possible actions in state $s$, irrespective of the chosen action $a$.

So both $a$ and $a'$ represent actions. $a$ is the current action, supplied on the LHS of the equation. $a'$ represents the iterator of a sum over all actions $[\forall a' \in \mathcal{A}(s)]$, only used in the calculation on the RHS. Sometimes you will see a completely different letter chosen, or some subscripting or other way to show these represent different actions.

It is also quite common to see $a$ representing current action, and $a'$ representing the next action (taken when in state $s'$). But that is not what is happening here.