# Does employment of engineered immediate rewards in RL introduce a non-linear problem to an agent?

Suppose we operate with state-action pairs called 'S', and a reward function R() as follows:

R(S1) <- 0
R(S2) <- 0
...
R(Sn) <- 1

so that

E(R(S)) << 1, i.e., non-zero rewards are very sparse


In that case a RL agent faces a task of accumulating as many as possible 1s in order to maximize

sum(gamma^i * Ri), i ->> infinity


Now consider immediate rewards introduced by one to make the agent better behaved:

R(S1) <- 0
R(S2) <- -1
R(S3) <- -10
R(S4) <- 0.1
...
R(Sn) <- 1

so that

E(R(S)) ->> -10, i.e., a simple average of random rewards is strongly negative.


Is it the case that in order to maximize the sum of discounted rewards the agent may choose to avoid the states with strongly negative rewards even at the cost of not getting positive rewards in some delayed states?

Is this really a non-linear problem to the model that approximates the agent's actions in that varying level of negative/positive immediate rewards (let's say, from -1 to 1, VS. from -100 to 100) may distract the agent from getting the delayed rewards which must be maximized under optimal policy in favor of maximizing immediate rewards, even if gamma ->> 1?

• Could you explain what you mean by 'linear'. Feb 8 '18 at 8:54
• @Valentas, I wanted to say in general that a slow change in the scale of immediate rewards, e.g., seq(from 0 to 100, by = 0.1) that a researcher may try affects the converged total discounted reward in a way that at some point the agent may sharply change its behaviour in favor of maximizing the immediate rewards, and the function of convergence which depends on the scale of the immediate rewards maybe be very non-linear. An an example, if one sets immediate rewards to be [-0.1;0.1] the agent's behavior converges to a perfect, while if the scale is [-1;1], convergence is not possible at all. Feb 8 '18 at 9:00
• Sorry for mistakes in my English. Feb 8 '18 at 9:00