I'd like to understand when using PG methods is more adequate than using DQN methods. Just to give a bit of background: I am currently using both APEX and R2D2 for my projects. Both work very well in achieving convergence across a variety of tasks, which I cannot say for PG based methods, such as PPO. One of the major differences of course being off-policy vs on-policy. Basically I think that having a replay buffer is incredibly useful, especially for tasks/games that do not have continuous rewards to guide the algorithm, such as MountainCar. Certain PG algorithms use off-policy correcting mechanisms, such as IMPALA using V-trace, etc, which helps, but is not truly allowing for off-policy behavior.
So to be more specific: If you don't require continuous action space, and you compare APEX vs PPO - at what point does it make more sense to use PPO? Are there any clear examples/situations where PPO would significantly outperform APEX? Anything where PPO would converge and APEX wouldn't?