I'm still relatively new to deep learning and am experiencing an issue that I can't seem to find a solution/explanation for. I've developed a DQN model in tensorflow, as described by DeepMind, and am attempting to train it on a game that I wrote myself. From what I can see, the only difference between the DeepMind model and the model I've implemented is that I am not using a target network. From my understanding of DQN, I don't think this has an impact on the issue that I am facing.
Issue: For some reason, the loss function seems to converge too quickly before the network has learned anything valuable.
Loss Functions: I've attempted several loss functions and optimizers with the same result. Initially, I attempted using an L2 loss function. Unfortunately, this caused my gradients to explode. I tried clipping the gradients between [-1,1]. Under this condition, the loss functions still converged but did so relatively quickly without the network learning anything valuable. I've looked more into the issue of exploding gradients and realized that the DeepMind describes loss clipping as a means of stabilizing the network. My understanding is that the paper essentially describes using a Huber loss function instead of an L2 loss. After implementing a Huber loss, the loss function still converges but does so more quickly.
My Impression: My impression of the issue is that the the network is falling into a local minimum and is unable to escape. However, from reading more into deep networks, it seems that falling to local minimums is not an issue given the large number of parameters present in the model.
Reward Function: Also I have some doubts about the reward function I am using and believe it may be causing this issue. The game's score depends on the length of the survival time. In order to account for this my reward function assigns reward by taking the length of survival time into account. The way this is done is by first collecting a game play iteration of experiences while playing the game. However, instead of the experiences noted as [state,action,reward,state], they are collected as [state,action,timestamp,state]. Note that these timestamps are collected in descending order. For example, if a game iteration survived a total of 15 seconds, then the earlier states are given timestamps closer to 15. Meanwhile later states, which lead to the inevitable termination, are given timestamps closer to zero. At this point, I apply a shifted and scaled sigmoid function to these timestamps to generate the reward for each experience. This way, states that lead to termination will be given values ranging between [-1,1) while all other states are given rewards of approximately 1. Note that I am still using experience replay to break the correlation between experiences.
If you have any insight into what maybe causing this, please let me know!