Why do trained RL agents still display stochastic "exploratory" behavior on testing data?

Question

I am training a PPO2 RL model using stable baselines. One thing I found is that a trained agent will still display some stochastic behavior on test data, as shown by the predict method of PPO2; the deterministic flag should be set to True to see deterministic (un-stochastic) behavior.

I understand that when training an RL agent, there is some amount of random exploration so that an agent can learn the optimal value-policy network. However, I thought that once an agent was trained, and used to act (predict) on new test data, the model parameters would be used and no exploration would take place. Therefore, when testing, I thought the agents actions and subsequent rewards would be deterministic (once the test data is static).

Why is the trained agent exploring on test data / Why is the trained agent still stochastic when predicting test data? To what degree is the trained agent stochastic (will it follow its model predictions 90% of the time and guess the other 10%)?

Answer 1

The PPO approach directly generates stochastic policies. Its output is some probability distribution function over the action space.

This is not the case for all RL algorithms, but is common for many Policy Gradient methods.

In some cases you may want this behaviour to continue. Paper-Scissors-Stone is a classic example of a game where a stochastic policy is optimal, and there are other examples from game theory.

Also, the agent many have learned on-policy (I'm not sure in case of PPO without looking it up), in which case the stochastic behaviour should match with expected value predictions. This is similar conceptually to using SARSA to generate an $\epsilon$-greedy policy. The value functions match following that policy, although if $\epsilon$ is low enough you may take a reasonable guess that a fully greedy policy is optimal.

To what degree is the trained agent stochastic (will it follow its model predictions 90% of the time and guess the other 10%)?

To the degree that the output of the policy is stochastic. It will always "follow its model predictions".

Switching deterministic on actually stops the agent from following the model, and typically will select the mode (highest probability density) of the action distribution.

Unlike SARSA, with Policy Gradient methods, there is not always access to a "greedy" policy that picks according to action values. So instead, your deterministic flag is likely picking greedily according to action probability.

In some cases (e.g. Actor-Critic) you may also have a value based estimator for V(s) or Q(s,a), and it could be used instead, but that is usually considered secondary to the policy function (in addition, using this on a large or continuous action space is very inefficient compared to a real-valued policy function)

Also note in some cases e.g. DDPG (which stands for Deep Deterministic Policy Gradients), a policy gradient method can use a deterministic policy with exploration added as a behaviour policy, making the algorithm off-policy. DDPG should not exhibit exploration during testing as PPO is for you. You may find DDPG behaves closer to your original expectations because of this.