0
$\begingroup$

What's the best approach to feed a bayesian description of an observed state to a Reinforcement Learning agent?

Brief context: I have an agent situated in an environment, which it perceives through a series of noisy sensors. Using Bayesian inference (outside of the RL agent) a number of sensed variables of interest are described in terms of their probability distributions, forming the state vector. As an example, think of the state as the position of an object in space, described at each instant with a multivariate Normal distribution.

The agent is tasked with gathering information about the state, meaning that its actions can impact the level of uncertainty used to describe the state. Following the previous example, if the location of the object is perceived through a movable camera, one of the actions could be to focus the camera on the object, lowering the variance in the probabilistic description of said location.

The state is assumed continuous, so a function approximation has to be used to retrieve the Q factor in each state. The set of actions is instead finite.

The reward function reflects this "variance minimizing" drive, as there's a penalty factor proportional to the variance of the observed state.

First question: is this actually doable with RL or am I missing something?

Second question: which choice might be better for the function approximator, knowing it has to deal with that probabilistic description of the state?

NOTE: The simpler the approach the better, as the training data is rather limited and definitely not enough for a Deep approach.

$\endgroup$

1 Answer 1

1
$\begingroup$

If the state uncertainty affects the decision making then there are possibly two avenues that you can take:

  • Incorporate the uncertainty in the state and formulate the problem as a Partially Observable Markov Decision Process (POMDP). An example of this is the RockSample domain. Some indicative links that could help you in your search: paper, code+visualization, code+examples+solvers

  • Another approach is to try to solve the exploration vs exploitation dilemma. Mainly if there are a number of different tasks (MDPs) and the agent doesn't know which task is facing it can maintain a belief distribution over tasks and use a solver to solve the task it believes is currently facing. This is the Bayes-Adaptive MDP framework.

You would need approximate methods to solve the problem. Aside the traditional POMDP/BAMDP solvers out there, there are also Neural Network approaches (usually model free RL) that solve these types of problems. However if you are not familiar with these problem formulations it is better to use the traditional methods first.

$\endgroup$
2
  • $\begingroup$ That's a very interesting paper and gave me a couple of idea on how to model the state in a - hopefully - useful manner. The gist of it is, as you mention, to also add the uncertainty in the states, so that in this case the model will try to model the probability distribution of the parameters of the observations, inspired by hierarchical bayesian models. Now I need to figure out how to deal with the continuous state for the tree search they adopt in the paper, but should not be too hard to wrap my head around. Thanks a lot! $\endgroup$
    – hypothe
    Aug 21, 2022 at 17:28
  • $\begingroup$ You are welcome! There are continuous state variations in the algorithms I posted above. If you think that your problem is computationally demanding to be solved with approximation methods then eventually you might want to consider Neural Network approaches: e.g. with latent variables or LSTMs, belief filtering etc. But if you are not familiar with POMDPs I totally recommend to start from the beginning: simple value iteration in a very small toy problem, then tree search methods with a harder problem etc. Good luck! $\endgroup$ Aug 22, 2022 at 3:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.