4
$\begingroup$

I have an agent which has a medium-sized, discrete set of actions $A$: $10<|A|<100$. The actions can be taken over an infinite horizon of 1 second per timestep $t$. The world is essentially pictures from a static camera and the states $S$ are the amount of animals detected on the picture (assuming perfect detection for simplicity). We can safely say that $max(S) < 200$. Each action $a$ is meant to lower the amount of detections because of dispersion or fear from the animals. The reward is the difference of detections from $s_{-1}$ to $s$.

Admittedly I am only a beginner in RL, so I haven't seen much more than MDP's. I believe that this isn't a Markov problem since the states aren't independent (past actions having an impact on the outcome of the current action). That being said, I'm wondering if there's a specific RL algorithm for this setup or if RL even is the right way to go ?

$\endgroup$
2
  • $\begingroup$ I think worth clarifying that you have a "scarecrow" set up where the camera is fixed and no actions can chnage the point of view of the camera (otherwise the agent could "stick its head in the sand"). Even actions such as setting off smoke might be problematic here . . . $\endgroup$ Aug 2, 2019 at 9:45
  • $\begingroup$ @Neil yes indeed the camera is static. Also the assumption of perfect detection implies that any partial occlusion is dealt with with other algorithms, sorry for the confusion. $\endgroup$
    – Mat
    Aug 2, 2019 at 10:36

1 Answer 1

3
$\begingroup$

Admittedly I am only a beginner in RL, so I haven't seen much more than MDP's. I believe that this isn't a Markov problem since the states aren't independent (past actions having an impact on the outcome of the current action).

If there are enough hidden variables, then this could be a real problem for you. A policy maps states to actions - and searching for an optimal policy will always map the same action to the same state. If this is some kind of automated scarecrow system, then the animals are likely to become habituated to the "best" action.

Two ways around that:

  • If the habituation is slow enough, you might get away with treating the environment as having the simple state that you suggest and have an agent which constantly learns and adapts to changes in what will trigger animals to leave. This would be an environment with non-stationary dynamics (the same action in the same state will, over time, drift in terms of its expected reward and next state). For a RL agent, this could just be a matter of sticking with a relatively high exploration rate and learning rate.

  • If the habituation is fast or adaptive, then you have to include some memory of recent actions used in the state representation. This will make the state space much larger, but is unavoidable. You could try to keep that memory inside the agent - using methods that work with Partially Observable MDPs (POMDPs) - it doesn't really make the problem any easier, but you may prefer that representation.

The reward is the difference of detections from $s_{-1}$ to $s$.

You need to re-think the reward system if habituation is fast/adaptive, as it will fail to differentiate between good and bad policies. All policies will end up with some distribution of detections, and all policies will have a mean total reward of 0 as a result. Your reward scheme will average zero in the long term for random behaviour, and also will average zero if the number of creatures remains at 200 the whole time, or is zero the whole time. Presumably having zero creatures would be ideal, whilst having 200 would be terrible - you need to be able to differentiate between those two scenarios.

You want to minimise the number of creatures in the area consistently, so a simple reward scheme is just negative of the number of visible animals, per time step. Maybe scale this - e.g. divide by 100, or by some assumed background rate where you would get -1 reward per time step if the number of animals is the same on average as if the agent took no action. You could measure this average using the camera over a few days when the agent is not present or simply not active. It doesn't need to be super accurate, teh scaling is just for convenience - you could think of an agent that gets a mean reward of -0.2 per time step is five times better than having no agent at all, whilst an agent that scores -1.5 per time step might be attracting creatures for all you know!

That being said, I'm wondering if there's a specific RL algorithm for this setup or if RL even is the right way to go ?

The problem does seem like a good match to RL. There are other ways of searching for good policy functions, such as genetic algorithms, that might also apply. However, you will still need to equip the agent with either continuous learning so it can re-find the best action as it changes, or with a memory of recent actions, depending on the speed of habituation. You may even need both as smart animals like birds or mammals can adapt in long and short term in different ways.

$\endgroup$
2
  • $\begingroup$ Interesting ! So yes indeed habituation is a real thing. However, I'm a bit confused by the mix of ideas here: can I keep the reward system I mentionned if I have continuous learning or must I change both the reward and the agent ? $\endgroup$
    – Mat
    Aug 2, 2019 at 17:40
  • 1
    $\begingroup$ Your reward system is bad over the long term. It is useless in a continuous environment as the average reard has to be zero. If the agent only operates in short term, or has to keep adapting, then perhaps your differential reward could still be useful. My gut feeling is that my suggestion will work better for you in all cases, and might be a more natural measure of success, but then i have never built a system like this, so I cannot say for sure. $\endgroup$ Aug 2, 2019 at 17:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.