what does $a'$ mean in the "combining" equation in Dueling DQN? (top of the page 5)

$$Q(s,a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \biggl( A(s, a; \theta, \alpha) - \frac{1}{N}\sum_{a'}^{N}A(s, a'; \theta, \alpha) \biggr)$$

Where there are $N$ actions to choose from;

  • $s$ is the incoming state (the input vector)
  • $a$ is the action taken? (the chosen action)
  • $a'$ I don't know what it represents in this context
  • $\theta$ represents the weights of the convolutional layers
  • $\alpha$ are the weights of the "Advantage stream" which outputs a vector
  • $\beta$ are the weights of the Value stream (which outputs a scalar)

Why not to simply use $a$ everywhere, why is $a'$ used in the average?


1 Answer 1


It is just a type of namespacing, because $a$ is already assigned the chosen action. There are two contexts of action being considered in the equation, so there needs to be a symbol for each context. Using $a'$ is an obvious choice as the letter $a$ is implicitly linked to representing an action already.

The sum over $a'$ is a sum over all possible actions in state $s$, irrespective of the chosen action $a$.

So both $a$ and $a'$ represent actions. $a$ is the current action, supplied on the LHS of the equation. $a'$ represents the iterator of a sum over all actions $[\forall a' \in \mathcal{A}(s)]$, only used in the calculation on the RHS. Sometimes you will see a completely different letter chosen, or some subscripting or other way to show these represent different actions.

It is also quite common to see $a$ representing current action, and $a'$ representing the next action (taken when in state $s'$). But that is not what is happening here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.