As you all know, DQN or DDQN are known for "unstable training".
Let's use the well known "CartPole". The agent has to balance the stick and gets a reward of +1 per frame. You can reach the 195 threshold with Cartpole-v0, but results will vary a lot. You will have a hard time to get this working until it is "nearly stable". Possible reasons are learning rate, batch size and so on...
If you master v0, switch to "Cartpole-v1" and I'm sure your "stable" system will fail again. You normally have to adapt parameters to make it working again. (just my experience)
But, there is something in the workflow of the algo, i don't understand:
for ep in range(num_episodes):
state = env.reset()
total = 0.0
done = False
while not done:
action = agent.get_action(state)
next_state, reward, done, info = env.step(action)
agent.remember(state, action, reward, next_state, done)
agent.train()
total += reward
state = next_state
ep_rewards.append(total)
You all have seen this workflow before, what's the problem here...?
1. We measure performance while we are training and moving the weights
Every agent.train() call does BATCH TRAINING and changes the weights. The "total" reward is calculated with a lot of different models, which one are we measuring?
2. In case of the cartpole example the episode ends (done) if it fails to balance
Some (the first) runs are very short - that leads to lesser training (inconsistent loop count). That means, if the agent performs bad, it does lesser training. If it works well - it trains a lot, does loop a lot until done and moves its weights away from the good policy and can get unstable.
3. If we save a model - which model are we really saving?
We effectively test a bunch of models and get some type of average performance, but what happens if we save the model after a good run? We can have a good run (high total reward) - but save a bad (the last) model? What weights are we really saving, i can't explain that in a way that makes sense, can you?
Now a simple improvement that solves all problems just by moving some code parts:
for ep in range(num_episodes):
state = env.reset()
total = 0.0
done = False
while not done:
action = agent.get_action(state)
next_state, reward, done, info = env.step(action)
agent.remember(state, action, reward, next_state, done)
total += reward
state = next_state
# the total result comes from a fixed model
# correct performance measures and saving ONE model are now possible
ep_rewards.append(total)
# train outside of while(done)
# every episode has now a constant number of train runs, 50 for example
for i in range(50):
agent.train()
After changing it i get a really stable performance at max values as you can see here:
Run 7 | Episode: 770 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 780 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 790 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 800 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 810 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 820 | eps 0.0 | total: 432.00 | ddqn True
Run 7 | Episode: 830 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 840 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 850 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 860 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 870 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 880 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 890 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 900 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 910 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 920 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 930 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 940 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 950 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 960 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 970 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 980 | eps 0.0 | total: 500.00 | ddqn True
Run 7 | Episode: 990 | eps 0.0 | total: 500.00 | ddqn True
Are my concerns valid?
Is this a valid improvement or do i miss something here?