# Deep Q Network gives same Q values and doesn't improve

I'm trying to build a deep Q network to play snake. I've run into an issue where the agent doesn't learn and its performance at the end of the training cycle is to repeatedly kill itself. After a bit of debugging, I figured out that the Q values the network predicts are the same every time. The action space is [up, right, down, left] and the network predicts [0, 0, 1, 0]. The training loss does go down over time, but it doesn't seem to make a difference. Here's the training code:

def train(self):
tf.logging.set_verbosity(tf.logging.ERROR)
self.build_model()
for episode in range(self.max_episodes):
self.current_episode = episode
env = SnakeEnv(self.screen)
episode_reward = 0
for timestep in range(self.max_steps):
env.render(self.screen)
state = self.screenshot()
#state = env.get_state()
action = None
epsilon = self.current_eps
if epsilon > random.random():
action = np.random.choice(env.action_space) #explore
else:
values = self.policy_model.predict(state) #exploit
action = np.argmax(values)
experience = env.step(action)
if(experience['done'] == True):
episode_reward += experience['reward']
break
episode_reward += experience['reward']
self.push_memory(Experience(experience['state'], experience['action'], experience['reward'], experience['next_state']))
self.decay_epsilon(episode)
if self.can_sample_memory():
memory_sample = self.sample_memory()
X = []
Y = []
for memory in memory_sample:
memstate = memory.state
action = memory.action
next_state = memory.next_state
reward = memory.reward
max_q = reward + (self.discount_rate * self.replay_model.predict(next_state)) #bellman equation
X.append(memstate)
Y.append(max_q)
X = np.array(X)
X = X.reshape([-1, 600, 600, 2])
Y = np.array(Y)
Y = Y.reshape([self.batch_size, 4])
self.policy_model.fit(X, Y)
food_eaten = experience["food_eaten"]
print("Episode: ", episode, " Total Reward: ", episode_reward, " Food Eaten: ", food_eaten)
if episode % self.target_update == 0:
self.replay_model.set_weights(self.policy_model.get_weights())
self.policy_model.save_weights('weights.hdf5')
pygame.quit()


Here's the network architecture:

    self.policy_model = Sequential()
self.policy_model.add(Conv2D(8, (5, 5), padding = 'same', activation = 'relu', data_format = "channels_last", input_shape = (600, 600, 2)))
rms = keras.optimizers.RMSprop(lr = self.learning_rate)
self.policy_model.compile(optimizer = rms, loss = 'mean_squared_error')


Here are the hyperparameters:

learning_rate = 1e-4
discount_rate = 0.99
eps_start = 1
eps_end = .01
eps_decay = 1e-5
memory_size = 100000
batch_size = 2
max_episodes = 1000
max_steps = 100000
target_update = 100


I've let it train for the full 1000 episodes and it's pretty bad at the end. Am I doing something wrong with the training algorithm?

EDIT: Forgot to mention that the agent receives a reward of 0.5 for going towards the food, 1 for eating the food, and -1 for dying

EDIT 2: Added updated code for models and hyperparameters

I can see you are using "plain" deepQ, consider using double deep Q or duelling, but go first on your current implementation.

First of all, most of the times the problem is a bug, can you provide the history of the scores?. It is important to understand if the agent is learning or the best solution it can find is killing itself to minimize loses.

As per your hyperparameters, I will suggest two increase gamma first and decay epsilon slower.

Another tweak will be your alpha, reduce it a bit, seems too high.

Anyway, I need more details to help you, for example, Are you reading the screen ?, if so, did you reduce the image to something manageable? Can you provide the full code?

• Full code is here: github.com/achandra03/snake-rl. I am reading the screen and I convert it to grayscale for manageability. – achandra03 Jan 2 '20 at 14:48
• Also forgot to mention that the scores seem to be alright in the beginning, by virtue of luck, but around 3/4 of the way through training, the agent just keeps killing itself. – achandra03 Jan 2 '20 at 18:04
• .- Are you sure that your environment is giving you the reward when you the agent dies ?, maybe the episode ends when you lost all your lives (like in some atari envoironments). In this case use a wrapper to get the episode done when agent lost the life. .- Are you sure that the action performed by the agent doesnt last 4 frames in the environment ?. In this case use a wrapper to ensure every action last only one frame. Sometimes in RL when the agent falls in a bad path it never recovers, but, I dont think this is the case, the first case is more likely to me. – Eduardo Di Santi Grönros Jan 8 '20 at 17:21
• The agent gets rewards, as I have it printing out at the end of each episode. Also, I only use 1 frame for each state as I think snake is simple enough to represent with one frame – achandra03 Jan 9 '20 at 17:38
• I was looking your hyper-parameters, seems ok, maybe the min epsilon is to high but don't think this is the problem. – Eduardo Di Santi Grönros Jan 11 '20 at 16:40

I was looking your hyper-parameters, seems ok, maybe the min epsilon is to high but don't think this is the problem. Your reward scheme is not the best. Going towards the food twice is the same as eating the food on time, so the return of a policy will vary a lot depending on how far is the food. In other words and only as an example, is better to go towards the food 2 times not eat the food and die than eat the food and then die. Probably the agent will need forever to play well.

Try to reduce your reward for moving to 0, will take more time time understand the objective of the game (eating) but will understand what to maximize.