I work for quite some time on a RL task which poses a surprising difficulty to the reinforcement learning agent to learn.
My technique is based on the Double DQN, with replay buffer, using recurrent NNs to evaluate Q, with e-greedy action selection (0.1 probability for taking a random action).
The most difficulty goes from the fact that the agent has to make quite a series of right actions in order to stumble upon a positive sparse reward. The number of actions it should make is just about 5 (and the action vocabulary contains just 3 actions), but it should make them in a specific environmental state, which also complicates things.
As a result, replay buffer can have zero positive rewards per 1024 visited states (very sparse).
I tried to shape intermediate rewards based on potential function, in order to inject some positive traits into the buffer.
My question is rather about intuition. Could you say the agent got stuck in a suboptimal policy?
From left to right:
- Q values of selected actions
- L2 loss
- average reward at the end of a trajectory.
Although the Q values and (not shown here) the moving sum of rewards stay positive, the final rewards as trajectory ends cannot break through the zero level (right pic). Anything above the zero level is a good result, and if it is above 1, the result is fantastically perfect.
My quess has been that the agent had found a 'loophole' for keeping the sum of intermediate and final rewards at maximum, while the final rewards are in fact negative.