1
$\begingroup$

Suppose we operate with state-action pairs called 'S', and a reward function R() as follows:

R(S1) <- 0
R(S2) <- 0
...
R(Sn) <- 1

so that

E(R(S)) << 1, i.e., non-zero rewards are very sparse

In that case a RL agent faces a task of accumulating as many as possible 1s in order to maximize

sum(gamma^i * Ri), i ->> infinity

Now consider immediate rewards introduced by one to make the agent better behaved:

R(S1) <- 0
R(S2) <- -1
R(S3) <- -10
R(S4) <- 0.1
...
R(Sn) <- 1

so that 

E(R(S)) ->> -10, i.e., a simple average of random rewards is strongly negative.

Is it the case that in order to maximize the sum of discounted rewards the agent may choose to avoid the states with strongly negative rewards even at the cost of not getting positive rewards in some delayed states?

Is this really a non-linear problem to the model that approximates the agent's actions in that varying level of negative/positive immediate rewards (let's say, from -1 to 1, VS. from -100 to 100) may distract the agent from getting the delayed rewards which must be maximized under optimal policy in favor of maximizing immediate rewards, even if gamma ->> 1?

$\endgroup$
3
  • $\begingroup$ Could you explain what you mean by 'linear'. $\endgroup$
    – Valentas
    Feb 8 '18 at 8:54
  • $\begingroup$ @Valentas, I wanted to say in general that a slow change in the scale of immediate rewards, e.g., seq(from 0 to 100, by = 0.1) that a researcher may try affects the converged total discounted reward in a way that at some point the agent may sharply change its behaviour in favor of maximizing the immediate rewards, and the function of convergence which depends on the scale of the immediate rewards maybe be very non-linear. An an example, if one sets immediate rewards to be [-0.1;0.1] the agent's behavior converges to a perfect, while if the scale is [-1;1], convergence is not possible at all. $\endgroup$ Feb 8 '18 at 9:00
  • $\begingroup$ Sorry for mistakes in my English. $\endgroup$ Feb 8 '18 at 9:00
1
$\begingroup$

Yes - Incentives matter.

Extreme rewards (positive or negative) will have extreme effects on a reinforcement learning (RL) agent.

Most contemporary RL system use deep learning which has the capacity to learn non-linear relationships.

$\endgroup$
2
  • $\begingroup$ I may have badly posed my question. Non linearity in how introducing intermediate rewards suddenly distracts the agent from learning optimal policy. A local minimum. I experienced this. Even small intermediate rewards can be an easier and faster-converging goal to obtain. My question was not exactly about non-linear approximations inside a NN. $\endgroup$ Jul 24 at 11:33
  • 1
    $\begingroup$ Rewards in a local minimum can be an issue. That is why most RL algorithms have a way to balance exploration and exploitation. $\endgroup$ Jul 24 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.