I am trying to learn a specific physical quantity (radiance) inside a 3D scene with Deep Q-Learning. Just to give a quick overview, my agent shoots rays inside the scene: the reward is the irradiance of the points hit. This means that the reward is given only when a light source is hit - only 1% of the times. This lead to a very sparse reward function.

My state is a tuple of the 3D spatial coordinates inside the scene, my actions are the possible discrete directions used by the agent to scatter rays. The q-values represent this physical quantity based on that specific action(/direction).


I expect the q-values to be higher in the first 10 actions, and then slightly decrease. This would reflect the physics of my system. When the training starts, this is actually the case:

enter image description here

After some episodes, the q-value of one action starts to spike, as seen in the figure below. This does not reflect the physics of the environment, for which the incoming radiance should be distributed over all the actions.

enter image description here

The trend lines for the q-values can be seen below. Each line is the q-value for a specific action. During the last iterations the q-values for two specific actions explode.

enter image description here
I know about target networks and experience replay (uniform and PER), but this did not solve my issue.

Are there any methods that I can try to deal with this problem? Is this a problem of vanishing/exploding gradient? Thanks


1 Answer 1


One guess is is that network isn't lucky enough to encounter (sufficient number of times) an example that leads to the reward. The issue is mentioned here pointing out that RL-agent might favor higher though incorrect estimates too early during learning. Link hypotheses that perhaps "step-annealing" could be better, instead of gradualy annealing the epsilon-greedy parameter.

In other words, every time you reduce the epsilon-greedy parameter, you reduce the network's chance to take a random action and actually encounter your (already rare by default) reward. Consider reducing the epsion-greedy parameter further in time, or maybe use a step function to anneal it.

There is also Hindsight experience replay. It tells you to concatenate to your input-vector the state in which you wish to ultimately end up (best-case scenario), and feed it into the network, for forward prop.

After it fails to encounter your reward, you take the state where the network ended up, and backpropagate as if that state was what you actually aimed for originally.

According to the above link you can also give it part of the state vector, if the exact destination state is unknown.

HER teaches your network to achieve smaller goals instead of "fixating" on the best-possible goal. Unlike the traditional approach "don't care how, just give me reward", HER describes how the final state looks like. Now, the network is able to learn how states relate to each other, and ultimately how they relate to the destination state the network was meant to go. (instead of mere "I failed" or "I succeeded")

Traditionally, your network won't get a chance to form an impression of the environment if encountered rewards are zero. With HER, you get such a chance even if the end goal is not visible on the horizon and rewards would be zero or uniform.

Here is a great blog about HER

Video of Hindsight experience replay


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.