I'm doing a course on reinforcement learning, and one of our tasks is to implement an agent on the Lunar lander continuous V2 environment from openAI gym. In order to solve the continuous problem, I used discretization for the number of actions, where the number of actions is supposed to be at least 8 (I chose 10). I used 3 methods: Deep Q Network, Double Deep Q Network and dueling, and I tried to solve it in two configurations: one regular and one with random noise (adding a zero-mean Gaussian noise with mean=0 and std = 0.05 to PositionX and PositionY observation of the location of the lander.)
The results I get: without noise: DQN reached avg. of 234, DDQN reached avg. of 229 and 3DQN reached avg. of 256. We can see that the 3DQN reaches much higher values in most episodes and the lowest values are still above 40. with noise: Total rewards per episode for DQN, DDQN, 3DQN train with random noise. We can see convergence at around 500 episodes for DDQN and 3DQN and around 800 episodes. The 3DQN is very stable and reaches high values faster than the DQN and DDQN.
Are these results reasonable? What are the main differences between the models that can explain these results? We also found that a network with about 2-3 hidden layers performs better than a deeper layer network. Does that makes sense?