I'd like to understand when using PG methods is more adequate than using DQN methods. Just to give a bit of background: I am currently using both APEX and R2D2 for my projects. Both work very well in achieving convergence across a variety of tasks, which I cannot say for PG based methods, such as PPO. One of the major differences of course being off-policy vs on-policy. Basically I think that having a replay buffer is incredibly useful, especially for tasks/games that do not have continuous rewards to guide the algorithm, such as MountainCar. Certain PG algorithms use off-policy correcting mechanisms, such as IMPALA using V-trace, etc, which helps, but is not truly allowing for off-policy behavior.

So to be more specific: If you don't require continuous action space, and you compare APEX vs PPO - at what point does it make more sense to use PPO? Are there any clear examples/situations where PPO would significantly outperform APEX? Anything where PPO would converge and APEX wouldn't?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.