I'm trying to implement a Double Dueling DQN on LunarLander and I'm facing an issue as my model is not learning so I'm trying to debug the graph and this leads me to a question regarding the prediction of the Q-value.
Q(s,a) = V(s) + A(s,a) − avg( A(s,a) )
I understand this equation in 2 different ways. Let's imagine we have a 2-actions environnent with
A = [0,6, 0,4],
V = 1,2 and we chose
action = 0.
Based on how I did it (let's call it Numpy's style), the Q-predicted is :
Q = 1.2 + [0,6, 0,4] - 0.5 = [1.3, 1.1]
But I can also understand it as if we compute only the value on the used action:
Q = 1.2 + 0.6 - 0.5 = [1.2 + 0.6 - 0.5, 0.4] = [1.3, 0.4]
This won't impact the error because the Q target will be the same as the Q-value except the index of the taken action but it can impact the prediction of action in the argmax as we fixed the n-action as only 1 "value" so actions are "linked".
Am I thinking the wrong way ? Is the Numpy's style right ?
Thanks for your light about it, Nicolas