I am new to reinforcement learning agent training. I have read about PPO algorithm and used stable baselines library to train an agent using PPO. So my question here is how do I evaluate a trained RL agent. Consider for a regression or classification problem I have metrics like r2_score or accuracy etc.. Are there any such parameters or how do I test the agent, conclude that the agent is trained well or bad.
Thanks
If your goal is optimal control, then you will want to measure the agent by how well it does at the task. You should use some aggregate measure of reward, such as total reward per episode (aka "return"), or mean reward per time step.
If you are working with a toy problem, or one designed so that it is easy to identify a maximum bound on the reward-based measure, then you can compare your agent against this known value. It is reasonable to expect a good agent to approach the maximum value.
In practice, many interesting problems do not have a known upper bound on reward totals or averages. For those problems, typically the best you can do is compare between agents. You can compare with:
A randomly acting agent. This would normally be just as a baseline, to show that the agent had learned something.
An automated agent using a simple action choice heuristic, which might be something natural or obvious in the given problem.
One or more humans on the same task
Other ML-trained agents including previous instances of the same agent
You will probably want to run multiple tests and average the results, if either the policy or environment are stochastic, in order to assess an agent with expected values as much as possible. It is also important, if you are using any off-policy technique such as DQN, to switch off any exploration during tests, to get a fair measure of how well the trained agent behaves (as opposed to how it behaves during training, which will be different).
If your agent is designed to continually learn and explore, and/or uses an on-policy approach, you can use results during training to assess it. For instance you can take a rolling average of total reward over the last N episodes or something similar. This is not a bad metric to monitor training, even for off-policy approaches, although for off-policy you will likely get an underestimate of performance compared to separate test runs.
There are other approaches and other metrics to assess an agent - e.g. how much experience, or how much computation the agent needs to learn to a certain level is often of interest.
If you want to "conclude that the agent is trained well or bad" for an optimal control task, then this assessment of total reward might be all you need. However, you can also look at loss metrics inside any neural networks - you would not do this in order to rank agents as "better" or "worse" but might do so in order to identify problems. These loss metrics are generally the same as supervised learning equivalents. So for instance in DQN, or for the "critic" part of PPO, you would be interested in whether the predicted value of any state matched the eventual value, and use MSE loss. For stochastic environments there is a caveat to this, that any collected data will be noisy so it is hard to tell the difference between a high loss due to variance in the data and high loss due to poor training or incorrect hyperparameters.
