The paper explaining 'Advantage Updating' as a method to improve Q-learning uses the following as its motivation.
Q-learning requires relatively little computation per update, but it is useful to consider how the number of updates required scales with noise or with the duration of a time step, At. An important consideration is the relationship between Q values for the same state, and between Q values for the same action. The Q values Q(x,u1) and Q(x,u2) represent the long-term reinforcement received when starting in state x and performing action u1 or u2 respectively, followed by optimal actions thereafter. In a typical reinforcement learning problem with continuous states and actions, it is frequently the case that performing one wrong action in a long sequence of optimal actions will have little effect on the total reinforcement. In such a case, Q(x,u1) and Q(x,u2) will have relatively close values. On the other hand, the values of widely separated states will typically not be close to each other. Therefore Q(x1,u) and Q(x2.u) may differ greatly for some choices of x1 and x2. Therefore, if the network representing the Q function makes even small errors, the policy derived from it will have large errors. As the time step duration dt approaches zero, the penalty for one wrong action in a sequence decreases, the Q values for different actions in a given state become closer, and the implied policy becomes even more sensitive to noise or function approximation error. In the limit, for continuous time, the Q function contains no information about the policy. Therefore, Q-learning would be expected to learn slowly when the time steps are of short duration, due to the sensitivity to errors, and it is incapable of learning in continuous time. This problem is not a property of any particular function approximation system; rather, it is inherent in the definition of Q values.
How do I mathematically prove that this effect occurs?
