Suppose we operate with state-action pairs called 'S', and a reward function R() as follows:

R(S1) <- 0
R(S2) <- 0
R(Sn) <- 1

so that

E(R(S)) << 1, i.e., non-zero rewards are very sparse

In that case a RL agent faces a task of accumulating as many as possible 1s in order to maximize

sum(gamma^i * Ri), i ->> infinity

Now consider immediate rewards introduced by one to make the agent better behaved:

R(S1) <- 0
R(S2) <- -1
R(S3) <- -10
R(S4) <- 0.1
R(Sn) <- 1

so that 

E(R(S)) ->> -10, i.e., a simple average of random rewards is strongly negative.

Is it the case that in order to maximize the sum of discounted rewards the agent may choose to avoid the states with strongly negative rewards even at the cost of not getting positive rewards in some delayed states?

Is this really a non-linear problem to the model that approximates the agent's actions in that varying level of negative/positive immediate rewards (let's say, from -1 to 1, VS. from -100 to 100) may distract the agent from getting the delayed rewards which must be maximized under optimal policy in favor of maximizing immediate rewards, even if gamma ->> 1?

  • $\begingroup$ Could you explain what you mean by 'linear'. $\endgroup$ – Valentas Feb 8 '18 at 8:54
  • $\begingroup$ @Valentas, I wanted to say in general that a slow change in the scale of immediate rewards, e.g., seq(from 0 to 100, by = 0.1) that a researcher may try affects the converged total discounted reward in a way that at some point the agent may sharply change its behaviour in favor of maximizing the immediate rewards, and the function of convergence which depends on the scale of the immediate rewards maybe be very non-linear. An an example, if one sets immediate rewards to be [-0.1;0.1] the agent's behavior converges to a perfect, while if the scale is [-1;1], convergence is not possible at all. $\endgroup$ – Alexey Burnakov Feb 8 '18 at 9:00
  • $\begingroup$ Sorry for mistakes in my English. $\endgroup$ – Alexey Burnakov Feb 8 '18 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.