1
$\begingroup$

I am teaching myself Deep Q Learning (and machine learning in general) using python and tensorflow to play Snake. My initial learning came mostly from this article. I have rewritten it in my own coding style and it is working well.

Now I am experimenting with different parameters and state information. The state for my game is an array of 11 values that looks like this:

0 - Danger 1 step ahead
1 - Danger 1 step on the right
2 - Danger 1 step on the left
3 - Snake is moving left
4 - Snake is moving right
5 - Snake is moving up
6 - Snake is moving down
7 - Food is on the left
8 - Food is on the right
9 - Food is on the upper side
10 - Food is on the lower side

All of these values are binary, either 0 or 1. To experiment, I wanted to switch out elements 3 through 6 with the constants I use for direction.

DIR_UP = 1
DIR_RIGHT = 2
DIR_DOWN = 3
DIR_LEFT = 4

So the state object would be an array of 8 elements, element 3 being non-binary:

0 - Danger 1 step ahead
1 - Danger 1 step on the right
2 - Danger 1 step on the left
3 - Snake movement direction
4 - Food is on the left
5 - Food is on the right
6 - Food is on the upper side
7 - Food is on the lower side

To me, this seems to have all the same information but in a different format, but the DQN apparently doesn't think so. After 150 training iterations on the first format, I get a competent snake that can regularly score 40 or more. To get the same scores on format 2 I needed to run 800 iterations with a slower epsilon decay. And in testing (without any randomness) it still takes indirect routes and needlessly crashes into the wall now and then, which I thought would have been weeded out.

So my main question is: what am I failing to understand about state format that makes my first so much more efficient than my second?

And sub-questions: are states supposed to be binary? What if I have to account for something like distance that can't be easily represented that way?

Thanks so much for you time and input!

$\endgroup$
0
$\begingroup$

You are increasing the state space by introducing non-binary options. This increases the complexity the model has to learn to reach the same level of performance.

In particular, you are creating a nested structure which makes it more difficult to assign reward credit to a specific option. The hierarchical options are "Should the snake move?" then "If yes, which direction?" make it slower to assign credit to the directional movement. A flat structure "Should the the snake move up?" allows credit to be more directly assigned to that decision.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.