I created a deep Q network to play snake. The code works fine, except for the fact that performance doesn't really improve over the training cycle. At the end, it's pretty much indistinguishable from an agent that takes random actions. Here's the training code:
def train(self): self.build_model() for episode in range(self.max_episodes): self.current_episode = episode env = SnakeEnv(self.screen) episode_reward = 0 for timestep in range(self.max_steps): env.render(self.screen) state = env.get_state() action = None epsilon = self.current_eps if epsilon > random.random(): action = np.random.choice(env.action_space) #explore else: values = self.policy_model.predict(env.get_state()) #exploit action = np.argmax(values) experience = env.step(action) if(experience['done'] == True): episode_reward += 5 * (len(env.snake.List) - 1) episode_reward += experience['reward'] break episode_reward += experience['reward'] if(len(self.memory) < self.memory_size): self.memory.append(Experience(experience['state'], experience['action'], experience['reward'], experience['next_state'])) else: self.memory[self.push_count % self.memory_size] = Experience(experience['state'], experience['action'], experience['reward'], experience['next_state']) self.push_count += 1 self.decay_epsilon(episode) if self.can_sample_memory(): memory_sample = self.sample_memory() #q_pred = np.zeros((self.batch_size, 1)) #q_target = np.zeros((self.batch_size, 1)) #i = 0 for memory in memory_sample: memstate = memory.state action = memory.action next_state = memory.next_state reward = memory.reward max_q = reward + self.discount_rate * self.replay_model.predict(next_state) #q_pred[i] = q_value #q_target[i] = max_q #i += 1 self.policy_model.fit(memstate, max_q, epochs=1, verbose=0) print("Episode: ", episode, " Total Reward: ", episode_reward) if episode % self.target_update == 0: self.replay_model.set_weights(self.policy_model.get_weights()) self.policy_model.save_weights('weights.hdf5') pygame.quit()
Here are the hyperparameters:
learning_rate = 0.5 discount_rate = 0.99 eps_start = 1 eps_end = .01 eps_decay = .001 memory_size = 100000 batch_size = 256 max_episodes = 1000 max_steps = 5000 target_update = 10
Here's the network architecture:
model = models.Sequential() model.add(Dense(500, activation = 'relu', kernel_initializer = 'random_uniform', bias_initializer = 'zeros', input_dim = 400)) model.add(Dense(500, activation = 'relu', kernel_initializer = 'random_uniform', bias_initializer = 'zeros')) model.add(Dense(5, activation = 'softmax', kernel_initializer = 'random_uniform', bias_initializer = 'zeros')) model.compile(loss='mean_squared_error', optimizer = 'adam')
For reference, the network outputs 5 values because of the 4 directions the snake can move, and 1 extra for taking no action. Also, instead of being a screenshot of the game like in a traditional DQN, I pass in a 400 dimensional vector as representation of the 20 x 20 grid that the game takes place in, though I believe that it should still work fine with this. The agent receives a reward of 1 for moving closer to the food or eating it and receives a reward of -1 if it dies. How can I get the performance to improve?