what is difference between the DDQN and DQN?

Question

I think I did not understand what is the difference between DQN and DDQN in implementation. I understand that we change the traget network during the running of DDQN but I do not understand how it is done in this code.

We put the self.target_model.set_weights(self.model.get_weights()) In implementation of DDQN and this is added when action is finished for DQN https://github.com/keon/deep-q-learning self.target_model.set_weights(self.model.get_weights()) is added to DQN in order to change the DQN to DDQN! But this happens when we are going out from running by break! There for there is no difference between them!

What is wrong in my mind? (Maybe the difference will be at test? Is this code for train and the test is done with setting the exploration rate=0 and then run this for just one episode with new weight we found? Is it right?

Therefore, what is difference between the presented DQN and DDQN of this link.

Answer 1

In particular, DQN is just Q-learning, which uses neural networks as a policy and use "hacks" like experience replay, target networks and reward clipping.

In original paper authors use convolutional network, which takes your image pixels and then fit it into a set of convolutional layers. However there are a couple of statistical problems:

1. DQN approximate a set of values that are very interrelated (DDQN solves it)
2. DQN tend to be overoptimistic. It will over-appreciate being in this state although this only happened due to the statistical error (Double DQN solves it)

$$Q(s,a) = V(s) + A(s,a)$$

By decoupling the estimation, intuitively our DDQN can learn which states are (or are not) valuable without having to learn the effect of each action at each state (since it’s also calculating V(s)).

We’re able to calculate V(s). This is particularly useful for states where their actions do not affect the environment in a relevant way. In this case, it’s unnecessary to calculate the value of each action. For instance, moving right or left only matters if there is a risk of collision

As @emilyfy said self.target_model.set_weights(self.model.get_weights()) - the updating of the target model.

Answer 2

From what I understand, the difference between DQN and DDQN is in the calculation of the target Q-values of the next states. In DQN, we simply take the maximum of all the Q-values over all possible actions. This is likely to select over-estimated values, hence DDPG proposed to estimate the value of the chosen action instead. The chosen action is the one selected by our policy model.

I looked through the codes and got confused too because this bit wasn't implemented. Then I realized they are commented out. The commented lines here would have selected the action for the next state using the current model and used the target model to get the Q-values for the selected action. They changed it in a commit some time ago, no idea why.

As for the code self.target_model.set_weights(self.model.get_weights()), it is the updating of the target model. The target model is supposed to have the same function as the policy model, but the DQN algorithm purposely separates them and updates it once in a while to stabilize training. It can be done once every certain number of steps, or in this case they seem to do it every episode.