I am trying to understand the underlying logic of Q learning (deep Q learning to be precise). At the moment I am stuck at the notion of future rewards.
To understand the logic, I am reviewing some of the present code samples. This one seemed quite interesting, so I went through it:
Here is the gist of the code that does the actual training of the underlying deep neural network:
def replay(self, batch_size): minibatch = random.sample(self.memory, batch_size) for state, action, reward, next_state, done in minibatch: target = reward if not done: target = (reward + self.gamma * np.amax(self.model.predict(next_state))) target_f = self.model.predict(state) target_f[action] = target self.model.fit(state, target_f, epochs=1, verbose=0) if self.epsilon > self.epsilon_min: self.epsilon *= self.epsilon_decay
In the 5th line of the code, (after the
if not done line) we are adding the discounted reward of the next step, to the present step, and setting it as the target reward of the executed action to be trained. So, the way I see it, we have the reward of the executed action, and discounted possible reward of the following action, combined.
As far as I understand, in each iteration, Q-learning algorithm predicts the future reward of next step (and next step only) using the machine learning technique in use (be it the CNN, DNN etc.). And we are multiplying the reward of next step (and that specific next step only) with discount rate, to make it less important than the immediate reward (with the ratio we specified). So, my question is, how does the algorithm takes even further steps (say, 5 steps) ahead into account?