As a newbie, for several months I browsed the internet hoping to find a user-friendly explanation of the Implicit Quantile Regression Network (IQN). But, it seems there is none at all.
In Reinforcement Learning, a DQN would simply output a Q-value for each action. This allows for Temporal Difference learning: linearly interpolating the current estimate of Q-value (of the currently chosen action) towards Q' - the value of the best action from the next state.
Quantile-Regression Network takes it a step further, outputting a range of values for each action. So if with DQN we had a vector of Q values, QR-DQN gives us a vector of sub-vectors of Q values. These Q-values are equally likely to occur when compared amongst each other (in that sub-vector).
For example, if we have 3 possible actions, QR-Network might output the following:
Q estimates for Action0: -15, -5, 120, 121 130 Q estimates for Action1: 20, 25, 200, 210 300 Q estimates for Action2: 18, 40, 41, 54 120
Once again, when working with QR-Network, we treat
-15, -5, 120, 121, 130 as having the same probability of occurring: in my case 0.2 or 20%
In other words, each action now has values sitting at the middles of 5 separate quantiles: at $\tau$ = 0.1, $\tau$=0.3, $\tau$=0.5, $\tau$=0.7, $\tau$=0.9
We can therefore say, that for the Action0:
- with probability of 20% values within [$-\infty$, -10] will occur
- with probability of 40% values within [$-\infty$, 57.5] will occur
- with probability of 60% values within [$-\infty$, 120.5] will occur
- with probability of 80% values within [$-\infty$, 121.5] will occur
- with probability of 100% values within [$-\infty$, $+\infty$] will occur
To conclude, with QR-Net we demand it to output correct sub-vectors, knowing that each Q-score must have the same probability as others in that sub-vector.
Now, how does IQN work?
From what I understood after checking the paper, we have to feed the network the $\tau$ itself (in my example either 0.1, 0.3, 0.5, 0.7 or 0.9),
Or should we instead feed it several random $\tau$, pulled from [0,1] range?
Say, we pulled-out
0.35, 0.05, 0.1, 0.2, 0.11, should we continue getting them, or should we stop once we taken out 5 of those $\tau$ values? Would that mean we have to do a forward-pass for every $\tau$ or can we use a vector of them all, at once?
As I understood, we have to feed this $\tau$ as an input, into the network, along with the state vector (concatenating).
Does IQN output only 1 value per action, or a vector of vectors, like QR-Network does?
The IQN paper gives a picture, but I am unable to grasp the intuition behind the 4th image:
Why does the above image show the $f$ function being used in DQN and in IQN but not in C51 or QR-DQN? Does that mean that the output is no longer a vector of vectors, but is just a vector, like in a usual DQN?
The gist of IQN is in the following paragraph, but I just can't grasp it fully. What's the intuition behind the cosine? I know it's a periodical function, why does it get computed 64 times with i=0 to i=64, with the same $\tau$ (for example $\tau=0.4$ etc)