I currently have a database recorded from human behavioural (observation, action, reward, next action) with 400k examples collected in real-life conditions.

I want to implement a kind of pretraining function for SAC algorithm from Stable Baselines 3 but it seems Imitation API (https://imitation.readthedocs.io/en/latest/) does not correspond exactly what i'm looking for for some reasons :

  • Imitation Learning is designed to reproduce a behaviour from (observation, action) couple and does not use recorded reward.
  • AIRL and GAIL does not support variable horizons (Actually, it seems true for almost all imitation algos)
  • SQIL can be unstable with continuous action spaces algorithms like SAC.

I read the paper "Accelrating inverse reinforcement learning with expert boostrapping" ( arXiv:2402.02608, Wu, Sanjiban 2024, URL : https://arxiv.org/abs/2402.02608). The proposed solution seems close of what i'm looking for. To make it short the concept is based on two principles :

  • Expert Replay Boostrapping (ERB)
  • Expert Q Boostrapping (EQB)

ERB seems easy to implement because it's no more than pre-filling the replay buffer with historical data but I don't know how to implement EQB... (Maybe training critic independantly before training actor ?)

Moreover, i read somwhere feedeing an RL algo with only "good" examples (especially without cases of illegal actions) could lead to instability...

Could someone help me on this point ?




Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.