I'm currently working on a collection of reinforcement algorithms: https://github.com/lhk/rl_gym
For deep q-learning, you need to calculate the q-values that should be predicted by your network. There are many strategies: monte-carlo, temporal-difference, TD(lambda), ... Basically you get a tradeoff, between the number of steps you look into the future, and the variance of your observations.
Instead of predicting the q values, it is also possible to predict advantages. Where
A(s, a) = Q(s, a) - V(s). So the advantage describes how much more than expected you get. It is discussed and motivated here.
For predicting the q values, you have to balance variance against the number of steps to look into the future. For the advantages, there is a method called generalized advantage estimation (GAE) which does this in a very neat way: https://arxiv.org/abs/1506.02438
I would like to predict those advantages, instead of q values. That is by no means a new idea, and apparently, advantage learning can outperform q-learning: http://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/photoz/.g/web/glossary/advantage.html
The above link is a very small abstract on advantage learning. The important part is:
Advantage learning [...] requires only that the A(x,u) advantages be stored
But how do I do that ? The GAE paper assumes that I can predict the value for every state. I need the values to calculate the advantages. As far as I can see, I have to options:
Predict only advantages and then somehow calculate the value from the advantage. As far as I can see, we need two out of: q-values, advantages, values. So if I don't want to predict the values, I have to predict the q-values. Which is the original problem. I read somewhere that the maximum advantage is the value, but that makes no sense to me and I can no longer find the link.
Predict both advantage and value. If I do this, implementing GAE and training the network to predict the advantages correctly would be simple. But what would I use as training target for the value ? If I use the GAE formulation for the advantages, it looks many steps into the future. It seems nonsensical to base the calculation of those advantages on a value function approximation that I train on one step lookaheads.
My question is not how to set up a function approximator, or what network types would be well suited for this. My question is: What are the target values for the value function, that I can feed to my function approximator. How do I actually train it ?