3
$\begingroup$

From left to right:

  • Maximum Q value for action selection (averaged)
  • Train error (averaged)
  • Reward from environment (averaged)

enter image description here

I run double Q-learning.

A behavioral policy is ε-greedy, ε constant decaying to 0.1.

I used learning rate decay as follows:

learning.rate = start_learn_rate / log(counter + 1), #decaying learning rate

optimizer = 'sgd'
start_learn_rate = 0.001

I observe degradation of the average reward after it hits its maximum, with a clear oscillation behaviour from high to low.

Is this chart typical to TD (Q) learning routine?

Could you advise on what parameters could be changed to stabilize the result?

Why the models behave like that? Don't they ought to expose a stable behaviour when Q is stabilized?

Environment, agent actions, and reward structure

My task is quite specific. I model sequences of actions applied to a process generating time series for the purpose of making something like predictions.

enter image description here

Observe a sine function (timeseries). Agent gets information about lagged values (and some engineered features) of that process from it's PAST. Another part of environment consists of a series of PAST actions (i.e., past 10 actions).

Actions are:

  • buy (green)
  • hold (black)
  • sell (red)

A return (basis for the reward) is generated when conditions meet:

  • buy -> sell (close buy position and immediately enter sell)
  • sell -> buy (close sell position and immediately enter buy)
  • buy -> hold (close buy and wait for next action)
  • sell -> hold (close sell and wait for next action)

The return is maximum (equals 2) when the agent keeps the series of one-type actions starting at hill and down to valley, and vice versa for other sign positions.

At this chart you could see that in the bottom plot the agent learned to guess the direction right in most cases, but the return is not maximized to the value of 2.

My goal is not just to observe the leraning statistics, but to find an optimal policy to be applied to an out of sample data, with further exploration and policy refinement.

My intuitive guess is that the learning rate is too high when theagent is close to optimality and is forced to escape the region:

enter image description here

UPDATE

I get almost perfect solution of the optimal policy in my toy task, however, the neural network is not quite stable, and I have to manually choose the moment to stop learning.

This is how the system evolved during training:

enter image description here

Again the optimality seems to be reached at some point (with later fluctuations).

The learning behaviour of the agent:

enter image description here

Note that after aprx. 30 000 iterations the anent starts to make clear sequences of actions leading to maximized reward.

At test (with greedy action selection) the behaviour is that:

enter image description here

And test trades are also pretty clear (not perfect though):

enter image description here

There are 2 points I want to improve:

  • Stopping criteria for learning
  • Stability of max-Q and averaged return (and sum of returns) during test.
$\endgroup$
  • 1
    $\begingroup$ Worth adding brief summary about your environment, because it could be important. Also, your third chart - is it the reward received during learning? Or is it a separate assessment running the optimal policy (i.e. without $\epsilon$, using pure greedy selection over Q)? $\endgroup$ – Neil Slater Jan 12 '18 at 15:33
  • 1
    $\begingroup$ Note if your goal is to receive maximum total reward during learning, you may prefer SARSA (or expected SARSA) over Q learning. If you just want to find an optimal policy, you should bear in mind that your learning graphs are not showing you the behaviour of the estimated optimal policy, but of the learning/exploring policy. $\endgroup$ – Neil Slater Jan 12 '18 at 15:36
  • $\begingroup$ @Neil Slater, thank you. I updated my question. This is the learning phase. However, I have infinite sample number (I can learn all the time) and am free to stop the learning at any time. The rewards are accumulated using exploration = 0.1. $\endgroup$ – Alexey Burnakov Jan 12 '18 at 16:02
  • 1
    $\begingroup$ When you say "Another part of environment consists of a series of PAST actions (i.e., past 10 actions)." do you mean that is stored in the environment only (i.e. it is hidden from the agent), or is it presented to the agent as part of the state. Is the state fully observed - i.e. the current state is a perfect predictor of [a probability distribution over] next state and reward? $\endgroup$ – Neil Slater Jan 12 '18 at 18:32
  • 1
    $\begingroup$ SARSA will often show better rewards during training, since its policy evaluation allows for the possibility of making mistakes due to the exploration. See SARSA's behaviour in cliff walking example: packtpub.com/mapt/book/big_data_and_business_intelligence/… - note this means that SARSA may be slower to learn the optimal policy, but may achieve more moderate results whilst working towards it. Whilst Q-learning will encourage riskier behaviour during learning $\endgroup$ – Neil Slater Jan 14 '18 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.