Deep Q network disregards input, giving identical output no matter the input state

Question

I've creating a very simple game, The board is an array of size 6. 0 is "empty cell" , 5 is "Goal", 8 is "player location" [8 0 0 5 0 0] for example means the agent needs to move 2 "right" to win.

hitting a "5" ends the game with reward +1 hitting a wall ends the game with reward -1 other moves have reward 0.

the "state" is simply the board, 6-long vector that can be 0\8\5

When i built a regular Q-Table based agent the game converged (to a good result) very fast.

The problem is that the DQN (Deep Q network) agent never converged. i sampled the network after a few thousand iterations and saw that the network for all inputs always has the same result.

The actual result keeps changing, each replay of the memory, but at each point if i ask to predict, no matter the input, the output will be "LEFT: X, RIGHT: Y". (X and Y are numeric values). the network ignores the input. any suggestions why?

Appendix: Hyperparams: batch-size = 64 memory = 100 (tried 1000 as well( gamma = 0.99 # discount rate epsilon_min = 0.05 epsilon_decay = 0.95 epsilon_start = 1.0 learning rate = 0.01 / 0.1 / 1.0

the network is ( i tried several):

        # Neural Net for Deep-Q learning Model
    model = K.models.Sequential()
    model.add(K.layers.Dense(8, input_dim=self.state_size, activation='relu'))
    model.add(K.layers.Dense(8, activation='relu'))
    model.add(K.layers.Dense(self.action_size, activation='linear'))

    model.compile(loss='mse',
                  optimizer=K.optimizers.Adam(lr=self.learning_rate))

Answer 1

The value function you want to learn should be relatively simple, but I would expect optimal action values to be quite similar between moving left or right, at least when that is away from a wall. Recall that action values measure the expected reward for taking the selected action, then following the policy from that point on.

So in state [0 0 8 5 0 0], the action value under an optimal policy for "move right" should be +1, but the action value for "move left" is not much different, at 0.9801 . . . because under an optimal policy the agent will move right twice afterwards when following the policy.

That means the neural network will have to learn fine detail differences between estimates. You have made this harder than it needs to be if you are using the representation as "raw" input to the neural network. Neural networks learn badly when the input number range is high. In addition your learning rates seem quite high.

My suggestions:

• Scale the inputs. You should have code that turns the state representation into neural network features. I'd suggest the neural network will function nicely with +1 for goal and -1 for agent, if you must work with this state representation.

• Try lower learning rates. A rate of $0.01$ is high for the Adam optimiser.

• Reduce discount factor $\gamma$ - that will make a clearer difference in this short episode-length environment between returns for faster and slower routes.

Answer 2

Solved. I changed the optimizer to SGD, it currently converge very slowly (6000 +- iterations), but it does converge