Timeline for What is "experience replay" and what are its benefits?
Current License: CC BY-SA 3.0
14 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 1, 2019 at 9:32 | comment | added | Neil Slater | @Gulzar: In practice it seems the Rainbow paper ignores all those details for the sake of simplicity. It just assigns the n-step return from the history table (even if the current target policy would not make the same action choices), and this is still better, empirically, than taking a 1-step return on the Atari games. I suspect for some environments that are more sensitive to policy changes - e.g. board games - that this would not work well at all. | |
| Feb 1, 2019 at 9:28 | comment | added | Neil Slater | @Gulzar: I mean those algorithms work nicely online, but you have to make compromises when using them offline. For instance, re-using a sampled trajectory introduces more sampling bias than re-using a single sampled transition (because each trajectory is a smaller fraction of all possible trajectories of the same length), and it is not possible to use eligibility traces between samples. When correcting for changes in target policy using importance sampling, longer trajectories are more likely to make zero-probability steps (in target policy) which contribute less to learning target Q values. | |
| Jan 31, 2019 at 23:24 | comment | added | Gulzar | @NeilSlater Why is it "harder to use multi-step learning algorithms"? What did you mean? | |
| Sep 26, 2018 at 12:49 | comment | added | StL | @Neil Slater, I've went through the Rainbow paper and I didn't see any special comments on using a special trick for combining experience replay and multi-step method. Also I've heard that multi-step method is originally impossible to combine with experience replay but why not just randomly pick n-consecutive experiences instead of 1 from experience replay but from the replay so that between each n-experiences, no correlations found? Isn't this multi-step experience replay? | |
| S Feb 15, 2018 at 12:08 | history | edited | Neil Slater | CC BY-SA 3.0 |
Import suggested paper reference suggested by @TQA
|
| S Feb 15, 2018 at 12:08 | history | suggested | TQA | CC BY-SA 3.0 |
Sorry for editting your answer. I can not comment due to the lack of reputations.
|
| Feb 15, 2018 at 11:38 | review | Suggested edits | |||
| S Feb 15, 2018 at 12:08 | |||||
| Nov 27, 2017 at 4:17 | comment | added | Shamane Siriwardhana | Yeah I went through the paper again. It also says this method can improve the off policy learning also . Because in Q learning with act according to epsilon-greedy policy but update values functions according to greedy policy. So when every time step our neural net parameters get updated by mini batch statistics which is more importantly not related to exact time step statistics but what happened before this also help to uncorrelated the data . | |
| Nov 26, 2017 at 21:45 | comment | added | Neil Slater | @ShamaneSiriwardhana: Yes I think you are right. It is the exact same data from the real trajectory, but instead of learning only from the most recent step, you save it in a big table and sample from that table (usually multiple samples, with a store of 1000s of previous steps to choose from). If you need more clarification, then maybe ask a question on the site. | |
| Nov 26, 2017 at 16:40 | comment | added | Shamane Siriwardhana | Let's say during the training we are in one state and we take an action according to epsilon-greedy policy and you end up in another state . So you get rewards , and the next state . Here the reward can be the score of the game and the states can be the pixel patterns in the screen . And then we take the error between our function aproximator and the value we got from the greedy policy again using already frozen function approximator . But with the experience replay when optimizing the approximator we take some random state action data set . Am I right ? | |
| Jul 19, 2017 at 12:32 | vote | accept | Ryan Zotti | ||
| Jul 19, 2017 at 11:26 | history | edited | Neil Slater | CC BY-SA 3.0 |
added 5 characters in body
|
| Jul 19, 2017 at 10:53 | history | edited | Neil Slater | CC BY-SA 3.0 |
added 127 characters in body
|
| Jul 19, 2017 at 7:25 | history | answered | Neil Slater | CC BY-SA 3.0 |