Effects of slipperiness in OpenAI FrozenLake Environment

Question

I am trying to wrap my head around the effects of is_slippery in the open.ai FrozenLake-v0 environment.

From my results when is_slippery=True which is the default value it is much more difficult to solve the environment compared to when is_slippery=False. It takes roughly 10K iterations to solve when is_slippery=True compared to roughly 150 iterations when is_slippery=False.

I used the same cross-entropy method for both of them.

Now my issue is trying to understand the implementation from the repository and how they were able to model steps in the environment in such a way to mimic slipperiness.

This is the implementation from the repository with the different ways steps are taken based on is_slippery.

for row in range(nrow):
            for col in range(ncol):
                s = to_s(row, col)
                for a in range(4):
                    li = P[s][a]
                    letter = desc[row, col]
                    if letter in b'GH':
                        li.append((1.0, s, 0, True))
                    else:
                        if is_slippery:
                            for b in [(a - 1) % 4, a, (a + 1) % 4]:
                                li.append((
                                    1. / 3.,
                                    *update_probability_matrix(row, col, b)
                                ))
                        else:
                            li.append((
                                1., *update_probability_matrix(row, col, a)
                            ))

Answer 1

According the documentation in that repo:

However, the ice is slippery, so you won't always move in the direction you intend.

That implies is_slippery increases the noise. Specifically, the transition matrix will be effected. It makes sense that your model takes longer to learn when there is an increase in noise between intended and actual action.

There are appears to be many hard-coded numbers in that example. It appears a person with deep knowledge of that specific environment added constants to account for the increase in noise.