One of the requirements of the Hindsight Experience Replay is supplying the DQN with a state and a goal (the desired end-state) that we hope to end up in:
<currState, goal> <-- For Hindsight Experience Replay;
<currState> <-- This would be an input for a usual DQN;
This paper allows to quickly learn when the rewards are sparse. In other words when the rewards are uniform for most of the time, with only a few rare reward-values that really stand out.
Question:
Let's say I want to have the player be killed by monsters in my game. Thus, my "goal state" must include a value of 0 for player's hit-points. However, the state-vector also includes his position (xyz coordinate), rotation vector, IDs of equipped items:
inputVec = <hp, x1,x2,x3, q1,q2,q3, chestID, handsID, headID, feetID>
I don't want to impose a specific position of a player, etc - I just want him dead. I only know what his 'hp' should be (should be zero), I don't care about the other values.
Therefore, I can't provide a perfectly well-defined goal vector - does this mean I can't use Hindsight Experience replay?
Edit:
my understanding is that components of currState
and goalState
must have identical components. We can't have these 2 vectors be of different sizes or store different things
Edit after accepting the answer:
As @lfelipesv mentioned, page 4 tells us:
We assume that every goal $g ∈ G$ corresponds to some predicate $f_g : S → \{0, 1\}$ and that the agent’s goal is to achieve any state s that satisfies $f_g(s) = 1$. In the case when we want to exactly specify the desired state of the system we may use $S = G$ and $f_g(s) = [s = g]$
The goals can also specify only some properties of the state, e.g. suppose that $S = \mathbb{R} ^2$ and we want to be able to achieve an arbitrary state with the given value of x coordinate. In this case $G = \mathbb{R}$ and $f_g((x, y)) = [x = g]$.