2
$\begingroup$

I implemented a self-critical policy gradient (as described here), for text summarization.

However, after training, the results are not as high as expected (actually lower than without RL...).

I'm looking for general guidelines on how to debug RL-based algorithms.


I tried :

  • Overfitting on small datasets (~6 samples) : I could increase the average reward , but it does not converge. Sometimes the average reward would go down again.
  • Changing the learning rate : I changed the learning rate and see its effect on small dataset. From my experiment I choose quite big learning rate (0.02 vs 1e-4 in the paper)
  • Looking at how average reward evolve as training (on full dataset) goes : Average reward significantly does not move...
$\endgroup$

2 Answers 2

1
$\begingroup$

For a deep dive into all sorts of possible issues that can come up in practical application of policy gradient, have a look at this paper:

A Closer Look at Deep Policy Gradients

Some common ways to improve policy gradient methods is:

  1. Confine the updates to a "trust region" near the current policy (see TRPO and PPO).
  2. Introduce a value network (called Critic in Actor-Critic) which can be used to subtract a baseline that reduces variance in policy-gradient learning.
$\endgroup$
0
$\begingroup$

The only resource I could find so far :

https://github.com/williamFalcon/DeepRLHacks


For my specific case, I made a few errors :

  • Frozen some part of the network that shouldn't be frozen
  • Wrong learning rate

Even if I could overfit a small dataset, it didn't mean anything : while training on the whole dataset, the average reward was not going up.

You should look for a reward going up.


I'm not accepting this answer as I believe it is not complete : it lacks general and systematic guidelines to debug a Reinforcement Learning algorithm.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.