From left to right:
- Maximum Q value for action selection (averaged)
- Train error (averaged)
- Reward from environment (averaged)
I run double Q-learning.
A behavioral policy is ε-greedy, ε constant decaying to 0.1.
I used learning rate decay as follows:
learning.rate = start_learn_rate / log(counter + 1), #decaying learning rate optimizer = 'sgd' start_learn_rate = 0.001
I observe degradation of the average reward after it hits its maximum, with a clear oscillation behaviour from high to low.
Is this chart typical to TD (Q) learning routine?
Could you advise on what parameters could be changed to stabilize the result?
Why the models behave like that? Don't they ought to expose a stable behaviour when Q is stabilized?
Environment, agent actions, and reward structure
My task is quite specific. I model sequences of actions applied to a process generating time series for the purpose of making something like predictions.
Observe a sine function (timeseries). Agent gets information about lagged values (and some engineered features) of that process from it's PAST. Another part of environment consists of a series of PAST actions (i.e., past 10 actions).
- buy (green)
- hold (black)
- sell (red)
A return (basis for the reward) is generated when conditions meet:
- buy -> sell (close buy position and immediately enter sell)
- sell -> buy (close sell position and immediately enter buy)
- buy -> hold (close buy and wait for next action)
- sell -> hold (close sell and wait for next action)
The return is maximum (equals 2) when the agent keeps the series of one-type actions starting at hill and down to valley, and vice versa for other sign positions.
At this chart you could see that in the bottom plot the agent learned to guess the direction right in most cases, but the return is not maximized to the value of 2.
My goal is not just to observe the leraning statistics, but to find an optimal policy to be applied to an out of sample data, with further exploration and policy refinement.
My intuitive guess is that the learning rate is too high when theagent is close to optimality and is forced to escape the region:
I get almost perfect solution of the optimal policy in my toy task, however, the neural network is not quite stable, and I have to manually choose the moment to stop learning.
This is how the system evolved during training:
Again the optimality seems to be reached at some point (with later fluctuations).
The learning behaviour of the agent:
Note that after aprx. 30 000 iterations the anent starts to make clear sequences of actions leading to maximized reward.
At test (with greedy action selection) the behaviour is that:
And test trades are also pretty clear (not perfect though):
There are 2 points I want to improve:
- Stopping criteria for learning
- Stability of max-Q and averaged return (and sum of returns) during test.