The PPO approach directly generates stochastic policies. Its output is some probability distribution function over the action space.
This is not the case for all RL algorithms, but is common for many Policy Gradient methods.
In some cases you may want this behaviour to continue. Paper-Scissors-Stone is a classic example of a game where a stochastic policy is optimal, and there are other examples from game theory.
Also, the agent many have learned on-policy (I'm not sure in case of PPO without looking it up), in which case the stochastic behaviour should match with expected value predictions. This is similar conceptually to using SARSA to generate an $\epsilon$-greedy policy. The value functions match following that policy, although if $\epsilon$ is low enough you may take a reasonable guess that a fully greedy policy is optimal.
To what degree is the trained agent stochastic (will it follow its model predictions 90% of the time and guess the other 10%)?
To the degree that the output of the policy is stochastic. It will always "follow its model predictions".
Switching deterministic on actually stops the agent from following the model, and typically will select the mode (highest probability density) of the action distribution.
Unlike SARSA, with Policy Gradient methods, there is not always access to a "greedy" policy that picks according to action values. So instead, your
deterministic flag is likely picking greedily according to action probability.
In some cases (e.g. Actor-Critic) you may also have a value based estimator for V(s) or Q(s,a), and it could be used instead, but that is usually considered secondary to the policy function (in addition, using this on a large or continuous action space is very inefficient compared to a real-valued policy function)
Also note in some cases e.g. DDPG (which stands for Deep Deterministic Policy Gradients), a policy gradient method can use a deterministic policy with exploration added as a behaviour policy, making the algorithm off-policy. DDPG should not exhibit exploration during testing as PPO is for you. You may find DDPG behaves closer to your original expectations because of this.