Reinforcement learning with sparse acting agent

Question

I'm working on a problem where the optimal policy involves the agent "doing nothing" most of the time, and "doing something" during rare critical moments. Is there any literature or best practices when it comes to solving problems like these where actions are quite rare? I tried changing the random action choice to choose inaction over action with a certain probability to bias the agent to do nothing, but that didn't really affect learning. I think part of the issue is that the agent only receives reward feedback when he acts, so this makes learning the optimal sparse action policy quite difficult. I've tried to avoid reward hacking (agent acting frequently and gathering many small rewards) by weighing the rewards in such a way that rewards given by infrequent but correctly timed action far out-weight rewards given by frequent, incorrectly timed action. But I'm not sure if that's a good solution, or even a solution at all. Currently using a dueling DQN network structure, but open to trying anything (policy gradients?). Any help would be much appreciated.

Answer 1

One option is to create a reward function to incentivize the behavior you want:

• Small positive reward for doing nothing.
• Large positive reward for doing the rare, correct action.
• Negative reward for taking action when no-action is correct.
• Negative reward for taking incorrect action.