How does Implicit Quantile-Regression Network (IQN) differ from QR-DQN?

Question

For several months I browsed the internet hoping to find a user-friendly explanation of the Implicit Quantile Regression Network (IQN). But, it seems there is none at all.

How does IQN differ from Quantile Regression Network, in plain language?

In Reinforcement Learning, a DQN would simply output a Q-value for each action. This allows for Temporal Difference learning: linearly interpolating the current estimate of Q-value (of the currently chosen action) towards Q' - the value of the best action from the next state.

Quantile-Regression Network takes it a step further, outputting a range of values for each action. So if with DQN we had a vector of Q values, QR-DQN gives us a vector of sub-vectors of Q values. These Q-values are equally likely to occur when compared amongst each other (in that sub-vector).

For example, if we have 3 possible actions, QR-Network might output the following:

Q estimates for Action0:  -15,  -5,  120,  121   130    
Q estimates for Action1:   20,  25,  200,  210   300
Q estimates for Action2:   18,  40,   41,   54   120

Once again, when working with QR-Network, we treat -15, -5, 120, 121, 130 as having the same probability of occurring: in my case 0.2 or 20%

In other words, each action now has values sitting at the middles of 5 separate quantiles: at $\tau$ = 0.1, $\tau$=0.3, $\tau$=0.5, $\tau$=0.7, $\tau$=0.9

We can therefore say, that for the Action0:

• with probability of 20% values within [$-\infty$, -10] will occur
• with probability of 40% values within [$-\infty$, 57.5] will occur
• with probability of 60% values within [$-\infty$, 120.5] will occur
• with probability of 80% values within [$-\infty$, 121.5] will occur
• with probability of 100% values within [$-\infty$, $+\infty$] will occur

To conclude, with QR-Net we demand it to output correct sub-vectors, knowing that each Q-score must have the same probability as others in that sub-vector.

Now, how does IQN work?

From what I understood after checking the paper, we have to feed the network the $\tau$ itself (in my example either 0.1, 0.3, 0.5, 0.7 or 0.9),

Or should we instead feed it several random $\tau$, pulled from [0,1] range? Say, we pulled-out 0.35, 0.05, 0.1, 0.2, 0.11, should we continue getting them, or should we stop once we taken out 5 of those $\tau$ values? Would that mean we have to do a forward-pass for every $\tau$ or can we use a vector of them all, at once?

As I understood, we have to feed this $\tau$ as an input, into the network, along with the state vector (concatenating).

Does IQN output only 1 value per action, or a vector of vectors, like QR-Network does?

The IQN paper gives a picture, but I am unable to grasp the intuition behind the 4th image:

Why does the above image show the $f$ function being used in DQN and in IQN but not in C51 or QR-DQN? Does that mean that the output is no longer a vector of vectors, but is just a vector, like in a usual DQN?

The gist of IQN is in the following paragraph, but I just can't grasp it fully. What's the intuition behind the cosine? I know it's a periodical function, why does it get computed 64 times with i=0 to i=64, with the same $\tau$ (for example $\tau=0.4$ etc)

Answer 1

Would like to thank Will Dabney and Georg Ostrovski (two of the paper's several authors), and Massimiliano Tomassoli (Simple-Machine-Learning blog owner) for helping me understand IQN.

To understand this algorithm, you will need to know:

• TD Learning
• Value Functions of Reinforcement Learning
• What a DQN is.
• What C51 is
• What a Cummulative Distribution Function is (CDF)
• What an Inverse CDF is.

Just like C51 or QR-DQN, IQN allows us to estimate an entire distribution of possible rewards if we take some action $a$. However, IQN doesn't output a distribution like those did. Instead, it outputs a single sample every time you ask it. It outputs that red dot you see on the right, of the bottom image.

The final output of IQN is 1 scalar value per action. In other words, just a single sample per action. We re-run IQN several times to get more samples for the "CurrentState".

IQN takes input at 2 different stages: 1) and 2)

1. First, the IQN takes a current state S (which is a vector), transforms it into another vector V (for example, of dimension 10).

2. now, we take any random scalar value called $\tau$ (drawn from a uniform [0,1] range), and feed that scalar value into function $\phi(\tau)$. It gives us a vector H, which has the same dimension as V

3. then, vectors V and H get combined via concatenation. In fact, we perform multiplication instead of concatenation if there won't be many forward layers further on.

4. At the end of this forward-pass, our IQN outputs an |A| dimensional vector, containing samples of the action-distributions we are trying to estimate. |A| is the other way to say the "number of actions". Notice: because IQN outputs a vector, and not a matrix (like QR-DQN does), we only get 1 scalar sample per action when performing a forward pass from the current State.

5. To get more samples, we have to return back to step 2) and re-run the forward pass with some another scalar $\tau$ once again sampled from [0,1] range. Or, of course, we can re-use our IQN, to run the needed number of such forward-passes in parallel, if hardware allows. No need to start all the way from step 1) because we only care about the end-layers where $\tau$ actually enters the system. "Any preceding layers" would be re-computed to the same values anyway.

When training, we evaluate "CurrentState+1" as well. We bootstrap our results from "CurrentState" towards "CurrentState+1" via usual TD-learning, just like a usual DQN would do. Don't forget, we will have to re-run forward pass several times (as step 5. has mentioned) both for "CurrentState" and for "CurrentState+1" to get several samples of the current and target distributions. This gives us a fairly good idea about what those two distributions look like.

6. According to the paper, in most cases re-running the forward pass 8 times (thus getting 8 samples) suffices for approximating both the $Z_{\tau}$ and our target distribution $Z$ (the distribution of the best action for "CurrState+1"). So we perform 16 forward passes in total: 8 for "CurrentState" and 8 for "CurrentState+1" to get an idea about those two distributions.

---

Why does feeding a random $\tau$ into the network work?

You have to understand that a well-trained IQN network represents an Inverse CDF itself. That is, given a request of some scalar 'amount' value (taken from a uniform range of 0 to 1), the Inverse CDF will output a value from the actual distribution, that sits at that amount.

Although distributions won't necessarily look like Bell-curves, have a look at this example:

---

Why do authors use cosine inside $\phi(\tau)$ ? I still have no idea, but the Appendix section of the paper states it seems to work the best. Perhaps it works good with ReLU.

---

I also got confused by $w_{ij}$ or $b_j$. To me it seemed there is a matrix of weights and a vector of bias values. ...Do we input an entire vector of different $\tau$ values at once?

The answer is no. Notice, authors used $j$ only to remind that forward pass has to run several times. In total, there is only i number of weights in $\phi(\tau)$ and 1 bias.

Therefore, we get some scalar $\tau$, and use it with all those weights. Basically $\phi(\tau)$ resembles a simple fully-connected layer, which only accept 1 scalar input value at a time. We will re-use this same layer for different $\tau$ scalar values.

Answer 2

Kari's answer is concise and very informative. However, the explanation for Equation 4 is slightly incorrect. (final section of the post)

$$\phi_j(\tau):=\text{ReLU}(\sum\limits^{n-1}_{i=0}\cos(\pi i\tau)w_{ij}+b_j)$$

$w_{ij}$ is indeed a matrix and $b_j$ is a vector.

It's better to interpret the equation by layers:

Let's say we have a single $\tau$, and we want to perform a single forward pass to get its value $Z_\tau(x_t,a_t)$.

• Input ($\tau$, $|\tau|=1$)

  We have a scalar $\tau$.

  $$\tau\sim U([0,1])$$

• Intermediate result ($I_i$, $|I|=n=64$)

  Expand the scalar $\tau$ to a vector by $\cos(\pi i\tau)$ for $i\in[0,n-1]$.

  $$I_i=\cos(\pi i\tau)$$

  The number 64 is used in the paper.

• Embedding of current $\tau$ ($\phi_j$, $|\phi|=256$)

  Use fully connected layer to expand $I$ to $\phi$ with $|W|=i\cdot j$ and $|b|=j$.

  $$\phi_j(\tau):=\text{ReLU}(\sum\limits^{n-1}_{i=0}I_i w_{ij}+b_j)$$

  The number 256 is the number of neurons of the layer after convolutions in DQN architecture.

• Then we conduct element-wise multiplication $\psi\odot\phi$ and continues pass it into DQN's last layer.

For multiple passes, we can simply treat all $\tau$ as a batch and run the forward pass in parallel.

The concept above is Universal Value Function Approximator (UVFA), while the cosine term is chosen since it results in good empirical results.