Recently I learned about Distributional approach to RL, which is a quite fascinating and break--through algorithm.

I have 2 questions:

What is it that makes it perform so much better during runtime than DQN? My understanding is that during runtime we will still have to select an action with the largest expected value. But to compute these expected values, we will now have to look at distributions of all possible actions at $x_{t+1}$, then select an action with a highest expected value. This would actually mean during extra work during runtime

What is the explanation for its faster converge than that of vanilla DQN? As I understand, the policy hasn't changed, we are still selecting the best action from state $x_{t+1}$, then use its best action's distribution for bootstrapping (adjusting) the distribution of our current state's best action.

Where does the Distributional part come into play and make the network be smarter about selecting the actions? (currently we still always select highest expected action as "the target distrib").

  • $\begingroup$ After learning more about this topic, "This would actually mean during extra work during runtime" - By now I realised this statement was not correct. During runtime we get the distribution at $x_t$; It's only the Training stage that "peeks into" x_{t+1} $\endgroup$ – Kari Jul 28 '18 at 21:18
  • $\begingroup$ Still curious about the second question though! $\endgroup$ – Kari Jul 28 '18 at 21:19

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.