1
$\begingroup$

Inverse reinforcement learning is about using expert trajectories to learn a reward function. Now the most successful method is Maximum Entropy Inverse Reinforcement Learning. But in that, you need a model-based reinforcement learning. But most of the practical world problems are model-free which is hard to calculate the state transition probabilities. So how can we use this inverse reinforcement learning in real-world problems?

$\endgroup$

closed as too broad by D.W., Stephen Rauch, Siong Thye Goh, Aditya, Toros91 Apr 6 '18 at 1:08

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

1
$\begingroup$

I think that the field has moved on from that paper. There is a trend to use the data from the expert to either "precondition" the agent or extracting a policy directly from the data. You can search for imitation learning or behavioral cloning. Some of these algorithms: Generative Adversarial Learning, DAGGER and Deep Q-learning from Demonstrations . AlphaGo used Supervised Learning as well to get to a good policy before getting trained in a RL setting. So instead of trying to recover a complicated reward function from the data you can use the above methods to get a good policy or initialize the parameters of your agent to more promising directions. Hope this helps!

$\endgroup$
  • $\begingroup$ but I feel like most of the real world problems cannot e model with normal deep reinforcement learning. Just because of the unavailability of reward function. $\endgroup$ – Shamane Siriwardhana Apr 5 '18 at 21:31
  • 1
    $\begingroup$ Few comments on your points: 1. Not always 2. Is not given but you can design it. For IRL do you intend to recover/estimate a reward function from data and then use model-free RL on that reward function? If thats the case, did you look carefully at the papers i sent you? One fundamental problem with IRL, even if you recover the reward successfully, there are might be multiple policies that lead to same behavior. So you might not be able to get the policy you want. Is there a specific reason that you want the maximum entropy one? $\endgroup$ – Constantinos Apr 7 '18 at 19:38
  • 1
    $\begingroup$ As i mentioned, even if you manage to recover the reward function perfectly (which i doubt in real world problems as you wouldn't know the ground truth) still your agent might have a very different behavior than the data you collected. Take a look at the papers I sent you and read at least the abstract and introduction sections. $\endgroup$ – Constantinos Apr 7 '18 at 19:46
  • 1
    $\begingroup$ yeah. The paper you sent GAIL is really good. I think it can be used to first recover the reward by demonstrations and then do model-free RL. Max- Entropy IRL normally needs to know the transition probabilities. But with the GAIL it is possible to do this in a model free way. BTW I believe these recovering reward function will be a good solution to build assisitve agents which can help in navigations or process optimization. Because we can use available data and also can introduce concepts like safety. What do you think ? $\endgroup$ – Shamane Siriwardhana Apr 8 '18 at 6:01
  • 1
    $\begingroup$ It seems that now researchers are more inclined to "precondition" their network, or having experts trajectories to affect the agent's training rather than recovering/approximating a reward function. And nowadays we have tons of data. Definitely the applications you are mentioning are possible! RL in real world applications is super hard right now, and as you mention using these data can eventually make it successful and safe -- which is really important. $\endgroup$ – Constantinos Apr 8 '18 at 17:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.