Timeline for Would Deep Q Learning work for a finite horizon problem?
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 28, 2019 at 19:14 | comment | added | Ufuk Can Bicici | Yes, actually exactly that. I just thought about it since my policies are not stationary. At each time step I have a different policy distribution. In common applications the policies are stationary; you use the same policy network over and over for each time step. But anyway, your answers were really helpful for me, thank you very much! | |
| Dec 28, 2019 at 19:07 | comment | added | Neil Slater | @UfukCanBicici: I cannot really tell how that would work from your comment. Are you suggesting having sub-optimisers that work with time horizon of 2 and 1, then using the 2-horizon value function to bootstrap the 3-horizon value function, and the 1-horizon function to bootstrap the 2-horizon one? Seems odd, but cannot think of a reason off top of my head why it would not work. I suspect it would not be very efficient and there may be a better way that i don't know however. Perhaps ask another question on the site? | |
| Dec 28, 2019 at 18:07 | comment | added | Ufuk Can Bicici | Would the following work then, based on the discussion so far; like in my previous comment: First optimizing $E[r_2]$ with respect to $\pi_2(a_2|s_2)$, using a vanilla policy gradient algorithm, without using actor critics or value functions and then optimizing $E[r_1 + V_2(s_2)]$ with respect to $\pi_1(a_1|s_1)$ by using the $V_2$ which we learn from the first optimization procedure. So there would be two separate neural networks for the two policies $\pi_1$ and $\pi_2$ in the end. | |
| Dec 28, 2019 at 17:51 | vote | accept | Ufuk Can Bicici | ||
| Dec 27, 2019 at 13:07 | comment | added | Neil Slater | @UfukCanBicici: Policy gradient methods should work, but any Actor-Critic variant will suffer a similar problem for estimating values. This answer would apply to those too - you could use a truncated Monte Carlo return in place of bootstrapping between value functions on different time steps. | |
| Dec 27, 2019 at 12:41 | comment | added | Ufuk Can Bicici | Oh, so I see that Q learning assumptions are about an infinte horizon case. Would policy gradients help me then? I think it is possible to maximize $V_2(s)$ wrt $\pi_2$ and then using them, frozen, for maximizing $V_1(s)$ wrt $\pi_1, as proposed here: rll.berkeley.edu/deeprlcoursesp17/docs/lec1.pdf | |
| Dec 27, 2019 at 12:22 | comment | added | Neil Slater | I think that there are Bellman equations that could be applied. However, the Bellman equations that are used for Q learning and relate value functions between time steps become much more cumbersome as you need to remove the reward beyond the horizon, so you end up with awkward sums like $v(s) = \mathbb{E}_{\pi}[R_{t+1} + v(S_{t+1}) - R_{t+4} |S_t = s]$ | |
| Dec 27, 2019 at 11:07 | comment | added | Ufuk Can Bicici | Many thanks for the detailed answer. Is the main problem in my case that the Bellman equation does not hold for me right? Is it only valid for problems with infinite time steps? I didn't exactly get what violates the Bellman equation in my case. | |
| Dec 27, 2019 at 10:03 | history | answered | Neil Slater | CC BY-SA 4.0 |