Temporal Difference Learning Getting Stuck

I'm trying to implement a temporal difference algorithm that learns the maximum revenue over a period of time using prices as the actions, inventory as the state, and revenues realized as the reward. The problem I'm having is that I can't seem to get it to converge on an optimal policy. It seems like it gets stuck at a revenue somewhere around 60% of optimization and then won't budge anymore. Are there some common pitfalls that might be causing this? I've tried playing with the rate of exploration a little bit, but that hasn't seemed to help it much.

EDIT: Ok, so I went back through everything, and it seems like the problem is that, when it stops exploring, it just keeps increasing the Q-values of states that it's already deemed the best in the past. So for example, it visits the price 5 at a certain state and time and gets a reward. Then, in the next several episodes, it continues to visit 5, and continues getting a reward, adding that reward to the Q-value until it's pretty high. At that point, even if it were to explore at the same state and time, the reward it gets isn't enough to overcome the inflated Q-value, so it just goes right back to 5 in the next episode. Here are the steps I'm trying to follow.

• I tried initializing the Q-values a couple different ways with no success. Let me see if I'm understanding the update process correctly: ∀ $i≤ t ∈ T$, ∀ $x_i$, $a_i$ Update all Q-Values according to their eligibility traces $Q_t^{k+1}(x_i, a_i) ← Q_i^k(x_i, a_i) + α(x_i^k,a_i^k)δ_t^ke_t^k(x_i,a_i)$ – user3704120 Jul 10 '15 at 1:06