What is "GOAL" in terms of Reinforcement Learning specified in these papers?

Question

I have a question regarding Reinforcement Learning. I've been reading the Horde and the UVFA paper extensively.

Take the Horde paper, there is this GVF, General Value Function Approximators which have a pseudo termination function and a pseudo reward. If I understand correctly, multiple demons with different behavioral policies, each of them having a GVF, spawned and off-policy learning takes place with GQ probability distribution. What is the pseudo termination function here? Is it a state?

Similar to this, there is this "goal" in UVFA paper. Take this paragraph for example, at the end of the second page:

A two-stream architecture, on the other hand, assumes that the problem has a factorized structure and computes its output from two components φ : S → R and ψ : G → R, that both map into an n-dimensional vector space of embeddings and output function h : R × R → R that maps two such embedding vectors to a single scalar regression output.

The novelty of this paper is that the value function generalizes not just over states but also goals, as introduced in the abstract. But I'm having hard time understanding the goal definition here. Again, is it a state that an agent wants to achieve?

So, can anyone give me a clue about the meaning behind these two concepts?

Thanks.

Answer 1

After some discussions with Kiuhnm in RL Group Discord channel, I understand it now. In the Horde paper it is easy, i.e. the goal might be state or a time. This sub-agent (called demon in the paper) might have a goal of running for 10 time steps, and you might have a reward function specific for it.

In the UVFA paper, the example was to define goals as a state and that's just for simplicity. Let's say you are in a room and your state is 2-dimensional position. Your goal maybe going to next room, and that means another 2-dimensional position. UVFA proposes that, the network will generalize over goals, and make your goal achieving easier in the following tasks.

That pretty much sums it up. Thanks to everyone who did give a thought.