One of the requirements of the Hindsight Experience Replay is supplying the DQN with a state and a goal (the desired end-state) that we hope to end up in:

<currState, goal>    <-- For Hindsight Experience Replay;
<currState>          <-- This would be an input for a usual DQN;

This paper allows to quickly learn when the rewards are sparse. In other words when the rewards are uniform for most of the time, with only a few rare reward-values that really stand out.


Let's say I want to have the player be killed by monsters in my game. Thus, my "goal state" must include a value of 0 for player's hit-points. However, the state-vector also includes his position (xyz coordinate), rotation vector, IDs of equipped items:

inputVec = <hp,  x1,x2,x3,  q1,q2,q3,  chestID, handsID, headID, feetID>

I don't want to impose a specific position of a player, etc - I just want him dead. I only know what his 'hp' should be (should be zero), I don't care about the other values.

Therefore, I can't provide a perfectly well-defined goal vector - does this mean I can't use Hindsight Experience replay?

Edit: my understanding is that components of currState and goalState must have identical components. We can't have these 2 vectors be of different sizes or store different things

Edit after accepting the answer:

As @lfelipesv mentioned, page 4 tells us:

We assume that every goal $g ∈ G$ corresponds to some predicate $f_g : S → \{0, 1\}$ and that the agent’s goal is to achieve any state s that satisfies $f_g(s) = 1$. In the case when we want to exactly specify the desired state of the system we may use $S = G$ and $f_g(s) = [s = g]$

The goals can also specify only some properties of the state, e.g. suppose that $S = \mathbb{R} ^2$ and we want to be able to achieve an arbitrary state with the given value of x coordinate. In this case $G = \mathbb{R}$ and $f_g((x, y)) = [x = g]$.


Using Hindsight Experience Replay you should be able to substitute achieved goals, so using the goal to be the final HP (in your case zero) can make the learning harder.

My idea would be to normalize the HP using something similar to: (1 - Current_HP/Max_HP) for the specific player. So it would be 0 when the player has its maximum HP, and 1 when the player is dead. The final goal would be always 1, and then you can calculate the achieved goal based on the normalized formula that I presented before (to substitute in the Hindsight Experience Replay algorithm)

<currState, goal>    <-- For Hindsight Experience Replay;
goal = 1
achieved_goal = 1 - current_hp/max_hp
  • $\begingroup$ Thanks! I am additionally unsure what to do with the remaining components of the goal vector (position, etc). I don't wish the NN to even pay attention to them, and instead to only care about the health-number. But I am still forced to supply at least something for position. Supposing I keep them zeros, I might confuse the network into thinking we are always discussing the World's zero coordinate. $\endgroup$ – Kari Oct 2 '18 at 6:26
  • $\begingroup$ Or do you reckon the network will learn to focus on hp and ignore whatever is in other components? In such a case, what to supply during runtime? Arbitrary (noisy) position? But we don't know where player will die yet $\endgroup$ – Kari Oct 2 '18 at 6:33
  • $\begingroup$ I don't know how important especifically is the other components of the goal vector for the task of killing a player in your game, but my approach would be to include them as a part of currState in your input vector. I would not set them to zero, and instead use the NN to care about them during training time. $\endgroup$ – lfelipesv Oct 4 '18 at 2:39
  • 1
    $\begingroup$ Not really... if you check the Hindsight Experience Replay paper on page 5, you are going to see in the algorithm that these two components are concatenated as the neural network input. I think it is because you are calling it "goalState", but it could be interpreted as "Goal" only... So, a concatenation: <currState, goal> $\endgroup$ – lfelipesv Oct 4 '18 at 11:18
  • 1
    $\begingroup$ It can be something like: currState = MonsterPosition, PlayerPosition, RotationVector, ItemIDs; goal = 1 ... input = <currState, goal> ... and during training you could change for the achieved_goal as I explained in my first answer $\endgroup$ – lfelipesv Oct 4 '18 at 11:21

I've got an answer from the Author:

The simplest solution would be to have 2 types of goals - a) specific hp and position; and b) specific hp, arbitrary position. I haven't tried anything like that though.

Based on the response I reason it might work if we do the following: use a placeholder value, perhaps for the last component of the input vector.
This way the DQN will notice that when this "last bit" is on, only health matters.

<hp,  x1,x2,x3,  q1,q2,q3,  chestID, handsID, headID, feetID,   placeholder0or1>

Edit: if you know a better way, please share it as an answer!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.