I'm trying to implement a temporal difference algorithm that learns the maximum revenue over a period of time using prices as the actions, inventory as the state, and revenues realized as the reward. The problem I'm having is that I can't seem to get it to converge on an optimal policy. It seems like it gets stuck at a revenue somewhere around 60% of optimization and then won't budge anymore. Are there some common pitfalls that might be causing this? I've tried playing with the rate of exploration a little bit, but that hasn't seemed to help it much.
EDIT: Ok, so I went back through everything, and it seems like the problem is that, when it stops exploring, it just keeps increasing the Q-values of states that it's already deemed the best in the past. So for example, it visits the price 5 at a certain state and time and gets a reward. Then, in the next several episodes, it continues to visit 5, and continues getting a reward, adding that reward to the Q-value until it's pretty high. At that point, even if it were to explore at the same state and time, the reward it gets isn't enough to overcome the inflated Q-value, so it just goes right back to 5 in the next episode. Here are the steps I'm trying to follow.