I'm trying to implement a Double Dueling DQN on LunarLander and I'm facing an issue as my model is not learning so I'm trying to debug the graph and this leads me to a question regarding the prediction of the Q-value.

Q(s,a) = V(s) + A(s,a) − avg( A(s,a) )

I understand this equation in 2 different ways. Let's imagine we have a 2-actions environnent with A = [0,6, 0,4], V = 1,2 and we chose action = 0.

Based on how I did it (let's call it Numpy's style), the Q-predicted is :

Q = 1.2 + [0,6, 0,4] - 0.5 = [1.3, 1.1]

But I can also understand it as if we compute only the value on the used action:

Q = 1.2 + 0.6 - 0.5 = [1.2 + 0.6 - 0.5, 0.4] =  [1.3, 0.4]

This won't impact the error because the Q target will be the same as the Q-value except the index of the taken action but it can impact the prediction of action in the argmax as we fixed the n-action as only 1 "value" so actions are "linked".

Am I thinking the wrong way ? Is the Numpy's style right ?

Thanks for your light about it, Nicolas


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.