I am running a Q-learning algorithm with a finite time horizon. Are 'optimistic initial conditions' still preferred if there is a possibility that some states will not be visited multiple times?

Wikipedia on Initial Q Values

  • $\begingroup$ Can you explain why you think optimistic initial conditions might not be preferred in the case you describe? Presumably each state will be visited multiple times over the course of training, even if they're never visited more than once per episode. $\endgroup$ – R Hill Oct 9 '16 at 0:22
  • $\begingroup$ If the Q-learner starts with extremely optimistic initial conditions, is it possible that the optimism of certain state/action pairs would not be eroded quickly enough to reflect their true value? I suppose the relative value would adjust immediately for the policy to avoid future negative pay-offs. $\endgroup$ – KT12 Oct 10 '16 at 18:23
  • $\begingroup$ Oh, right. So if you have a state, $s$, with two actions, $a_1$ and $a_2$, and both have optimistic initial values, you're worried that on the first training episode, it'll discount $Q(s, a_1)$, but not to its true value, and on the second training episode, it'll discount $Q(s, a_2)$ to a value higher than $Q(s, a_1)$, and it'll then never visit $a_1$ from $s$ again, and never allow $Q(s, a_1)$ to reach its true value? This seems like a legitimate concern, but Q-learning is supposed to converge on an optimal policy, not on a true value function. $\endgroup$ – R Hill Oct 11 '16 at 9:16
  • $\begingroup$ If you have a finite time horizon is your objective to gain most reward within the time horizon, or to have most optimal policy by the end? If the former, an on-policy algorithm like SARSA (or Expected SARSA) may be a better choice, unless for some reason you are restricted to consider Q-Learning. $\endgroup$ – Neil Slater May 9 '17 at 19:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.