Q-Learning: Target Network vs Double DQN

Question

I am having a hard time understanding difference between Target Network and Double DQN

From this blog:

Target Network generates the target-Q values that will be used to compute the loss for every action during training. The target network’s weights are fixed, and are frequently but by small amounts updated towards the primary Q-networks values.

Double DQN: instead of taking the max over Q-values when computing the target-Q value for our training step, we use our primary network to chose an action, and our target network to generate the target Q-value for that action

That sounds very similar to me, and the equation makes me lost too :(

qTarget = r + γQ(  s’,  argmax(Q(s’,a,ϴ),ϴ’)  )

From whaat I know, Target Network Approach allows us to do the following:

1. using Online Network, get scores for each action as the response for the current state. Pick the most appropriate action $a$ and take a note of its score $q$
2. using that action, "travel" to the next state and get scores from Target Net for even further actions.
3. Select the most appropriate action from step 2) and call it $A$. also, take a note at its score $Q$
4. Punish OnlineNetwork for any difference between $q$ and $Q$.

Note: the gradient flows only through $a$ component of the "output vector" $\{a, b, c, d ...\}$

Please correct me if these steps are wrong

Answer 1

Ok, it's simple!

Just the "Target network approach":

1. Select an item from Memory Bank
2. Using Target Network, from $S_{t+1}$ determine the index of the best action $A_{t+1}$ and its Q-value
3. Do corrections as usual

The "double DQN approach"":

1. Select an item from Memory Bank
2. Using Online Network, from $S_{t+1}$ determine the index of the best action $A_{t+1}$.
3. Using Target Network, from $S_{t+1}$ get the Q-value of that action.
4. Do corrections as usual using that Q-value

As seen from 3, the Online Network gives us the action that it thinks is highest. However, we let the Target Network have the final say on the score of such an action.

Were we to let Target Network select a best action, it could have very well selected some other index.