Suppose we operate with state-action pairs called 'S', and a reward function R() as follows:
R(S1) <- 0 R(S2) <- 0 ... R(Sn) <- 1 so that E(R(S)) << 1, i.e., non-zero rewards are very sparse
In that case a RL agent faces a task of accumulating as many as possible 1s in order to maximize
sum(gamma^i * Ri), i ->> infinity
Now consider immediate rewards introduced by one to make the agent better behaved:
R(S1) <- 0 R(S2) <- -1 R(S3) <- -10 R(S4) <- 0.1 ... R(Sn) <- 1 so that E(R(S)) ->> -10, i.e., a simple average of random rewards is strongly negative.
Is it the case that in order to maximize the sum of discounted rewards the agent may choose to avoid the states with strongly negative rewards even at the cost of not getting positive rewards in some delayed states?
Is this really a non-linear problem to the model that approximates the agent's actions in that varying level of negative/positive immediate rewards (let's say, from -1 to 1, VS. from -100 to 100) may distract the agent from getting the delayed rewards which must be maximized under optimal policy in favor of maximizing immediate rewards, even if gamma ->> 1?