What is the purpose of reward threshold in OpenAI Gym?

Question

I've seen that OpenAI Gym environments can be registered with an optional reward threshold (reward_threshold) which represents:

The reward threshold before the task is considered solved

How does this value affect the learning process? Or does one have to manually compare the reward obtained in each episode with the reward_threshold and stop the learning process if it surpasses it?

Answer 1

reward_threshold (float) – Gym environment argument, the reward threshold before the task is considered solved [1]

Just from that one sentence definition, it sounds like a total reward that e.g. an agent must earn, before the task is complete, and so ends.

If this were for example a task for the cartpole agent to stay upright/vertical, it might be formulated as the number of frames, so 1 frame = 1 point reward and if reward_threshold = 200, the agent must balance the pole for 200 frames to succeed.

Have a look at the example of cartpole on the OpenAI Gym website:

while True:
    candidate_model = model.symmetric_mutate()
    rewards = [run_one_episode(env, candidate_model, False) for _ in range(5)]
    reward = np.mean(rewards)
    if reward >= env.spec.reward_threshold:
        print "Reached reward threshold!"
        rewards2 = [run_one_episode(env, candidate_model, False) for _ in range(env.spec.trials)]
        if np.mean(rewards2) >= env.spec.reward_threshold:
            break
        else:
            print "Oops, guess it was a fluke"

So the agent runs for 5 episodes (for _ in range(5)) and each episode returns an award. We compute the mean reward over 5 episodes (reward = np.mean(rewards)) and then introduce the desired control flow, based on that result.

In this case, they use the default reward_threshold from the environment (env.spec.reward_threshold), print a success message, and do a final check to see if the agent really has learnt something consistent or not, by checking once again on a new set of episode (for _ in range(env.spec.trials)).

Answer 2

I did some digging in the gym codebase, and at least as of v.0.18.0, gym itself doesn't appear to be using reward_threshold at all (as opposed to max_episode_steps, which is used to compute the Done signal when stepping in the environment).

So one would have to manually access this field from the env if they wanted to use it.