Recently I learned about Distributional approach to RL, which is a quite fascinating and break--through algorithm.

I have 2 questions:

What is it that makes it perform so much better during runtime than DQN? My understanding is that during runtime we will still have to select an action with the largest expected value. But to compute these expected values, we will now have to look at distributions of all possible actions at $x_{t+1}$, then select an action with a highest expected value. This would actually mean during extra work during runtime

What is the explanation for its faster converge than that of vanilla DQN? As I understand, the policy hasn't changed, we are still selecting the best action from state $x_{t+1}$, then use its best action's distribution for bootstrapping (adjusting) the distribution of our current state's best action.

Where does the Distributional part come into play and make the network be smarter about selecting the actions? (currently we still always select highest expected action as "the target distrib").

  • $\begingroup$ After learning more about this topic, "This would actually mean during extra work during runtime" - By now I realised this statement was not correct. During runtime we get the distribution at $x_t$; It's only the Training stage that "peeks into" x_{t+1} $\endgroup$ – Kari Jul 28 '18 at 21:18
  • $\begingroup$ Still curious about the second question though! $\endgroup$ – Kari Jul 28 '18 at 21:19

This is meant to be a comment, but I can't comment since I have insufficient reputation.

As for the second question, intuitively speaking, instead of taking a scalar value for an action, which initially, may be highly inaccurate and noisy, taking a distribution instead would be more accurate. I'd recommend https://flyyufelix.github.io/2017/10/24/distributional-bellman.html which explains the intuitive reason for using a distribution

In terms of convergence, actually, there is no guarantee of convergence. In the paper, however, explains that for distributional DQN to guarantee to converge, the gamma-contraction must be satisfied, which would be true if you measure the the distance between the distributions using wasserstein distance, but it would be impractical to try to minimize that distance, so distributional DQN uses cross entropy instead which you can find the gradients of, and perform backpropagation....etc

You may be interested in "Distributional Reinforcement Learning with Quantile Regression" https://arxiv.org/pdf/1710.10044.pdf which aims to improve the original distributional DQN algorithm


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.