1
$\begingroup$

I am new to reinforcement learning agent training. I have read about PPO algorithm and used stable baselines library to train an agent using PPO. So my question here is how do I evaluate a trained RL agent. Consider for a regression or classification problem I have metrics like r2_score or accuracy etc.. Are there any such parameters or how do I test the agent, conclude that the agent is trained well or bad.

Thanks

$\endgroup$
1
$\begingroup$

If your goal is optimal control, then you will want to measure the agent by how well it does at the task. You should use some aggregate measure of reward, such as total reward per episode (aka "return"), or mean reward per time step.

If you are working with a toy problem, or one designed so that it is easy to identify a maximum bound on the reward-based measure, then you can compare your agent against this known value. It is reasonable to expect a good agent to approach the maximum value.

In practice, many interesting problems do not have a known upper bound on reward totals or averages. For those problems, typically the best you can do is compare between agents. You can compare with:

  • A randomly acting agent. This would normally be just as a baseline, to show that the agent had learned something.

  • An automated agent using a simple action choice heuristic, which might be something natural or obvious in the given problem.

  • One or more humans on the same task

  • Other ML-trained agents including previous instances of the same agent

You will probably want to run multiple tests and average the results, if either the policy or environment are stochastic, in order to assess an agent with expected values as much as possible. It is also important, if you are using any off-policy technique such as DQN, to switch off any exploration during tests, to get a fair measure of how well the trained agent behaves (as opposed to how it behaves during training, which will be different).

If your agent is designed to continually learn and explore, and/or uses an on-policy approach, you can use results during training to assess it. For instance you can take a rolling average of total reward over the last N episodes or something similar. This is not a bad metric to monitor training, even for off-policy approaches, although for off-policy you will likely get an underestimate of performance compared to separate test runs.

There are other approaches and other metrics to assess an agent - e.g. how much experience, or how much computation the agent needs to learn to a certain level is often of interest.

If you want to "conclude that the agent is trained well or bad" for an optimal control task, then this assessment of total reward might be all you need. However, you can also look at loss metrics inside any neural networks - you would not do this in order to rank agents as "better" or "worse" but might do so in order to identify problems. These loss metrics are generally the same as supervised learning equivalents. So for instance in DQN, or for the "critic" part of PPO, you would be interested in whether the predicted value of any state matched the eventual value, and use MSE loss. For stochastic environments there is a caveat to this, that any collected data will be noisy so it is hard to tell the difference between a high loss due to variance in the data and high loss due to poor training or incorrect hyperparameters.

$\endgroup$
  • $\begingroup$ Hi, I am training an agent for optimal control. My episode here is for 10 hrs in a day, and and my agent takes an action every 15 mins. So when I test it for one episode, I am not sure whether the action taken is the best action or not because, the action taken might help it in future because reward is formulated for entire episode, so I feel the actions taken may be un-explainable humanly. Thats why I want to resort to some metrics which will tell whether the agent is trained well enough. $\endgroup$ – cvg Oct 30 at 13:24
  • $\begingroup$ @cvg: This answer already addresses that. You need to assess using aggregate measures of reward - e.g. the sum of all rewards from an episode. It is not possible to assess the agent's individual action choices, other than that they lead to a good expected reward for the episode as a whole. $\endgroup$ – Neil Slater Oct 30 at 13:28
  • $\begingroup$ Hi !! My total rewards are increasing during training and converging, but intuitively I feel agent could have taken better actions. So, I am still in a doubt that even though my cumulative reward is increasing and converging, is the agent trained well enough. In supervised learning accuracy of around 80% for a classification problem tells us that there is some scope of betterment of a model. How to do that for a RL problem. How do we evaluate against a different model trained for same objective ? $\endgroup$ – cvg Oct 31 at 11:05
  • $\begingroup$ @cvg: I have answered that already. There is no equivalent to that 80% accuracy score to assess the model independently of the environment. You evaluate against a different model by measuring the expected total reward using both models, using the environment. Higher expected total reward is better. This is already written in the answer. $\endgroup$ – Neil Slater Oct 31 at 11:44
  • $\begingroup$ Hi, thank you for your answer and comments. $\endgroup$ – cvg Oct 31 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.