1
$\begingroup$

I'm trying to implement a Double Dueling DQN on LunarLander and I'm facing an issue as my model is not learning so I'm trying to debug the graph and this leads me to a question regarding the prediction of the Q-value.

Q(s,a) = V(s) + A(s,a) − avg( A(s,a) )

I understand this equation in 2 different ways. Let's imagine we have a 2-actions environnent with A = [0,6, 0,4], V = 1,2 and we chose action = 0.

Based on how I did it (let's call it Numpy's style), the Q-predicted is :

Q = 1.2 + [0,6, 0,4] - 0.5 = [1.3, 1.1]

But I can also understand it as if we compute only the value on the used action:

Q = 1.2 + 0.6 - 0.5 = [1.2 + 0.6 - 0.5, 0.4] =  [1.3, 0.4]

This won't impact the error because the Q target will be the same as the Q-value except the index of the taken action but it can impact the prediction of action in the argmax as we fixed the n-action as only 1 "value" so actions are "linked".

Am I thinking the wrong way ? Is the Numpy's style right ?

Thanks for your light about it, Nicolas

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.