1
$\begingroup$

I created a deep Q network to play snake. The code works fine, except for the fact that performance doesn't really improve over the training cycle. At the end, it's pretty much indistinguishable from an agent that takes random actions. Here's the training code:

def train(self):
        self.build_model()
        for episode in range(self.max_episodes):
            self.current_episode = episode
            env = SnakeEnv(self.screen)
            episode_reward = 0
            for timestep in range(self.max_steps):
                env.render(self.screen)
                state = env.get_state()
                action = None
                epsilon = self.current_eps
                if epsilon > random.random():
                    action = np.random.choice(env.action_space) #explore
                else:
                    values = self.policy_model.predict(env.get_state()) #exploit
                    action = np.argmax(values)
                experience = env.step(action)
                if(experience['done'] == True):
                    episode_reward += 5 * (len(env.snake.List) - 1)
                    episode_reward += experience['reward']
                    break
                episode_reward += experience['reward']
                if(len(self.memory) < self.memory_size):
                    self.memory.append(Experience(experience['state'], experience['action'], experience['reward'], experience['next_state']))
                else:
                    self.memory[self.push_count % self.memory_size] = Experience(experience['state'], experience['action'], experience['reward'], experience['next_state'])
                self.push_count += 1
                self.decay_epsilon(episode)
                if self.can_sample_memory():
                    memory_sample = self.sample_memory()
                    #q_pred = np.zeros((self.batch_size, 1))
                    #q_target = np.zeros((self.batch_size, 1))
                    #i = 0
                    for memory in memory_sample:
                        memstate = memory.state
                        action = memory.action
                        next_state = memory.next_state
                        reward = memory.reward
                        max_q = reward + self.discount_rate * self.replay_model.predict(next_state)
                        #q_pred[i] = q_value
                        #q_target[i] = max_q
                        #i += 1
                        self.policy_model.fit(memstate, max_q, epochs=1, verbose=0)
            print("Episode: ", episode, " Total Reward: ", episode_reward)
            if episode % self.target_update == 0:
                self.replay_model.set_weights(self.policy_model.get_weights())
        self.policy_model.save_weights('weights.hdf5')
        pygame.quit() 

Here are the hyperparameters:

learning_rate = 0.5
discount_rate = 0.99
eps_start = 1
eps_end = .01
eps_decay = .001
memory_size = 100000
batch_size = 256
max_episodes = 1000
max_steps = 5000
target_update = 10

Here's the network architecture:

model = models.Sequential()
model.add(Dense(500, activation = 'relu', kernel_initializer = 'random_uniform', bias_initializer = 'zeros', input_dim = 400))
model.add(Dense(500, activation = 'relu', kernel_initializer = 'random_uniform', bias_initializer = 'zeros'))
model.add(Dense(5, activation = 'softmax', kernel_initializer = 'random_uniform', bias_initializer = 'zeros')) 
model.compile(loss='mean_squared_error', optimizer = 'adam')

For reference, the network outputs 5 values because of the 4 directions the snake can move, and 1 extra for taking no action. Also, instead of being a screenshot of the game like in a traditional DQN, I pass in a 400 dimensional vector as representation of the 20 x 20 grid that the game takes place in, though I believe that it should still work fine with this. The agent receives a reward of 1 for moving closer to the food or eating it and receives a reward of -1 if it dies. How can I get the performance to improve?

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.