I have started learning reinforcement learning and have few doubts regarding model based and model free methods.
Is it possible to model an environment that is stochastic in nature?
Is it because its difficult to model such environment we use model free methods?
Is it possible to model an environment that is stochastic in nature?
Yes, a model of a stochastic environment can be one of:
A distribution model, that outputs a probability distribution for next state and reward, given current state and action. If you are reading Sutton & Barto, or similar work which uses the function $p(s',r|s,a)$, then if you can implement this function for the whole environment, it means that you have access to a distribution model.
A sampling model, that outputs a single reward and next state, given current state and action, with the same probability of any outcome as the real environment. If you can implement an accurate simulation of an environment, then you have a sampling model.
If you want to use approaches such as Dynamic Programming, which work with expected values, then this is much easier with a distribution model, and in that case you need an accurate model from the start (otherwise Dynamic Programming may converge to a non-optimal policy).
Is it because its difficult to model such environment we use model free methods?
Not really, there is no special difficulty in modelling stochastic environments. For instance, if your environment is a dice game, it is just a matter of implementing the rules and a random number generator for the dice to create a sampling model. A distribution model is usually straightforward for basic dice rules, such as rolling a die to see how many steps you can take.
However, independently of how random the environment is, complex environments can become hard to model. Distribution models may require a lot of maths to calculate all possibilities, so sampling models (simulations) are easier. For instance, a card game is relatively easy to implement in simulation when you track the deck contents. But a distibution model for it is more complex, as you have to track what has already been played or can be figured out from other observations.
Many environments are too complex to model. For example, they may involve real world physics but don't include enough measurements to establish the full state. For instance, when an agent flies a drone it will be affected by air turbulence but cannot directly observe it. Chaotic effects such as turbulence are very hard to model, and gaining real world experience is likely going to be more accurate than any model based on a physics engine, no matter how hard you try to code one. Similarly, visualising the real world, or navigating human social environments can be very hard to model accurately.
When deciding whether or not to use a model-free method, there needs to be a cost analysis. Even in complex environments you may prefer to use a model-based method:
The advantage of a model is that it allows you to safely explore without taking a real action, and also in some cases it may be a lot faster to query the model than to take an action and wait for a result in the real world. In the time it takes for a real action to resolve a computer may be able to check 10, 100, 1000 or more simulated actions from its model.
The disadvantages of a model are:
In some cases, the real environment is fast, safe and reliable enough that there's not much to gain by using a model. These cases include a lot of the toy problems used to study learning, where the environment is actually simulated (which is a form of model) but kept separate from the agent (so despite this, the agent is still technically model-free).
