Q learning - how to use experience replay, when playing against other agent?

Question

I am currently trying to create a tic tac toe q learning neural network to introduce me to reinforcement learning, however it didn't work so I decided to try a simpler project requiring a network to train against static data rather than another neural network. This lead to me following the guidelines from this website - http://outlace.com/rlpart3.html

however after programming this, the simple version works half the time, this is the version without experience replay. on some runs of the program the game will be played correctly, others it just moves back and forth wehn doing test runs. When trying to implement experience replay to complete the harder version. the program will just constantly get itself into a loop of going back and forth when testing

i have a limit of 100 batches in which a batch is what the neural network is trained on. I am wondering whether this is an appropriate amount, or if there could be any common problems with implementing experience replay that i may have made.

My current perspective of what experience replay is: 1. run the program 2. after each turn, the data you used to train the network on, gets saved into a batch 3. when you have reaches x(100) amount of batches, pick one out and train on it. 4. overwrite the oldest batch with the new batches that come in.

If anyone could let me know where I have gone wrong, or if there is any feedback about the experience replay or the quality of the question please let me know and i would be very grateful.

EDIT: Another question I have in terms of training a neural network against a neural network, is that do you train it against a completely separate network that trains itself,m or do you train it against a previous version of itself. And when training it against the other neural network, do you turn the epsilon greedy down to make the opposing neural network not use any random moves.

Answer 1

I am pretty sure you have to:

wait for enough data in the memory (10 000 entries, etc), then take out a minibatch (for example, 64 random elements), run backprop once, using this minibatch.

You then take an average of resulted gradients and finally, correct your network with that averaged gradient.

After,

• put the elements back in the memory (where you've taken them from)

Samples that have high priority are likely to be used in training many times. Reducing the weights on these oft-seen samples basically tells the network, "train on these samples, but without much emphasis; they'll be seen again soon."

• or throw it out, substituting it with new memories instead.

Then keep playing the game, adding say, another 10 examples before doing another weights-correction session after assembling a batch of 64 random elements.

I refer to the "Session" meaning a sequence of backprops, where the result is the average gradient used to finally correct the network.

EDIT: Another question I have in terms of training a neural network against a neural network, is that do you train it against a completely separate network that trains itself, or do you train it against a previous version of itself. And when training it against the other neural network, do you turn the epsilon greedy down to make the opposing neural network not use any random moves.

Consider using just one network. Let's say our memory bank contains several elements:

...
{...}
{stateFrom, takenActionIndex, takenActionQval, immediateReward, stateNext }  <-- a single element
{...}
{...}
{...}
...

When using each element in your memory during the Correction session (one element after the other), you need to:

1. pick an element. As seen, it contains stateFrom, taken action (action X), qvalue of action X, reward you received, state that it led to.
2. Run forward prop as if you were in "next state" (mentioned in that element). Get its best [action Y, qvalue]. In fact, your action Y doesn't have to be the action with highest q-value. Instead, it could be the "epsilon-greedy decided action" from the "next state" - you would then have SARSA instead of Q-learning
3. obtain delta by which the "qvalue of action X" differs from the immediate reward + "qvalue of action Y".
4. get deltas of several other elements that you've chosen from the bank
5. Using each delta, compute a gradient tensor for your network's weights. Average those gradients into a single tensor. Punish your network's weights with this tensor.

I've intentionally circumvented 1 thing - you don't actually store takenActionQval, because they might become obsolete by the time you fetch its element from MemoryBank. You have to re-compute these scores during backprop.

---

Therefore, you are training against the version of your network from the similar but previous correction session.

Notice, you don't store ["next state", action Y] because by the time you select it to train (maybe you didn't select it for several minibatches), the network might have a different q-value for that action Y.

You could also copy your network to the second network (target network), but only, say, every 200 timesteps. You would still proceed to punish your network for any differences with the target network in the meantime, after every 30 timesteps.

Notice, the intuition of why this works is because: the Q-values are sort-of "flowing" from finish back to the beginning, by a little with every new journey. And you are always updating the current state towards the successor state. And train the successor state against its own (even further) successor.