1
$\begingroup$

I've creating a very simple game, The board is an array of size 6. 0 is "empty cell" , 5 is "Goal", 8 is "player location" [8 0 0 5 0 0] for example means the agent needs to move 2 "right" to win.

hitting a "5" ends the game with reward +1 hitting a wall ends the game with reward -1 other moves have reward 0.

the "state" is simply the board, 6-long vector that can be 0\8\5

When i built a regular Q-Table based agent the game converged (to a good result) very fast.

The problem is that the DQN (Deep Q network) agent never converged. i sampled the network after a few thousand iterations and saw that the network for all inputs always has the same result.

The actual result keeps changing, each replay of the memory, but at each point if i ask to predict, no matter the input, the output will be "LEFT: X, RIGHT: Y". (X and Y are numeric values). the network ignores the input. any suggestions why?

Appendix: Hyperparams: batch-size = 64 memory = 100 (tried 1000 as well( gamma = 0.99 # discount rate epsilon_min = 0.05 epsilon_decay = 0.95 epsilon_start = 1.0 learning rate = 0.01 / 0.1 / 1.0

the network is ( i tried several):

        # Neural Net for Deep-Q learning Model
    model = K.models.Sequential()
    model.add(K.layers.Dense(8, input_dim=self.state_size, activation='relu'))
    model.add(K.layers.Dense(8, activation='relu'))
    model.add(K.layers.Dense(self.action_size, activation='linear'))

    model.compile(loss='mse',
                  optimizer=K.optimizers.Adam(lr=self.learning_rate))
$\endgroup$
  • 1
    $\begingroup$ what is the input to the network? $\endgroup$ – hh32 Oct 24 at 15:18
  • $\begingroup$ I'm guessing the network input is literally just the array as described. Seems an odd way to represent the state. There are only 30 states possible in this game, but choosing that state representation it is possible to represent millions of non-valid states. However, it should be colvable with the state representation as given. $\endgroup$ – Neil Slater Oct 24 at 15:52
  • $\begingroup$ Can you recommend a better way to model this game for the NN? $\endgroup$ – Guy Barash Oct 24 at 16:33
  • $\begingroup$ @GuyBarash: There are a few ways, and you should ask a separate question if you want details. But to start, a two element array with position of agent and goal would be simple. Scale it to between -1 and 1, or one-hot-encode it for input to the NN. It's a toy problem, and sometimes it is OK to make these harder by selecting difficult representation to work from. So personally I'd leave it as-is and go on to the next idea unless you are really keen to play with representations to see the difference. $\endgroup$ – Neil Slater Oct 24 at 17:38
1
$\begingroup$

The value function you want to learn should be relatively simple, but I would expect optimal action values to be quite similar between moving left or right, at least when that is away from a wall. Recall that action values measure the expected reward for taking the selected action, then following the policy from that point on.

So in state [0 0 8 5 0 0], the action value under an optimal policy for "move right" should be +1, but the action value for "move left" is not much different, at 0.9801 . . . because under an optimal policy the agent will move right twice afterwards when following the policy.

That means the neural network will have to learn fine detail differences between estimates. You have made this harder than it needs to be if you are using the representation as "raw" input to the neural network. Neural networks learn badly when the input number range is high. In addition your learning rates seem quite high.

My suggestions:

  • Scale the inputs. You should have code that turns the state representation into neural network features. I'd suggest the neural network will function nicely with +1 for goal and -1 for agent, if you must work with this state representation.

  • Try lower learning rates. A rate of $0.01$ is high for the Adam optimiser.

  • Reduce discount factor $\gamma$ - that will make a clearer difference in this short episode-length environment between returns for faster and slower routes.

$\endgroup$
0
$\begingroup$

Solved. I changed the optimizer to SGD, it currently converge very slowly (6000 +- iterations), but it does converge

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.