Admittedly I am only a beginner in RL, so I haven't seen much more than MDP's. I believe that this isn't a Markov problem since the states aren't independent (past actions having an impact on the outcome of the current action).
If there are enough hidden variables, then this could be a real problem for you. A policy maps states to actions - and searching for an optimal policy will always map the same action to the same state. If this is some kind of automated scarecrow system, then the animals are likely to become habituated to the "best" action.
Two ways around that:
If the habituation is slow enough, you might get away with treating the environment as having the simple state that you suggest and have an agent which constantly learns and adapts to changes in what will trigger animals to leave. This would be an environment with non-stationary dynamics (the same action in the same state will, over time, drift in terms of its expected reward and next state). For a RL agent, this could just be a matter of sticking with a relatively high exploration rate and learning rate.
If the habituation is fast or adaptive, then you have to include some memory of recent actions used in the state representation. This will make the state space much larger, but is unavoidable. You could try to keep that memory inside the agent - using methods that work with Partially Observable MDPs (POMDPs) - it doesn't really make the problem any easier, but you may prefer that representation.
The reward is the difference of detections from $s_{-1}$ to $s$.
You need to re-think the reward system if habituation is fast/adaptive, as it will fail to differentiate between good and bad policies. All policies will end up with some distribution of detections, and all policies will have a mean total reward of 0 as a result. Your reward scheme will average zero in the long term for random behaviour, and also will average zero if the number of creatures remains at 200 the whole time, or is zero the whole time. Presumably having zero creatures would be ideal, whilst having 200 would be terrible - you need to be able to differentiate between those two scenarios.
You want to minimise the number of creatures in the area consistently, so a simple reward scheme is just negative of the number of visible animals, per time step. Maybe scale this - e.g. divide by 100, or by some assumed background rate where you would get -1 reward per time step if the number of animals is the same on average as if the agent took no action. You could measure this average using the camera over a few days when the agent is not present or simply not active. It doesn't need to be super accurate, teh scaling is just for convenience - you could think of an agent that gets a mean reward of -0.2 per time step is five times better than having no agent at all, whilst an agent that scores -1.5 per time step might be attracting creatures for all you know!
That being said, I'm wondering if there's a specific RL algorithm for this setup or if RL even is the right way to go ?
The problem does seem like a good match to RL. There are other ways of searching for good policy functions, such as genetic algorithms, that might also apply. However, you will still need to equip the agent with either continuous learning so it can re-find the best action as it changes, or with a memory of recent actions, depending on the speed of habituation. You may even need both as smart animals like birds or mammals can adapt in long and short term in different ways.
