I've built a game with an agent moving around trying to collect gold (+10 reward) and it dies when it hits a wall (-100, terminal condition) and -1 for any other step that is not gold nor wall. My action space is 4 (U, D,R,L) and my state is the sight he has. [More details in the end of the message]
The problem is it improves for a few hundred\thousands (depending on the model) iterations and then it drops to into a mode it simply runs into a wall, never improving from there.
This is my model:
self.model = K.models.Sequential() self.model.add(K.layers.Dense(128, input_shape=(self.observation_space,), activation="relu")) self.model.add(K.layers.Dropout(0.1)) self.model.add(K.layers.Dense(64, activation="relu")) self.model.add(K.layers.Dropout(0.1)) self.model.add(K.layers.Dense(self.action_space, activation="linear")) optimizer = K.optimizers.Adam(lr=0.01) self.model.compile(loss="mse", optimizer=optimizer)
I've tried debugging it, and encountered the following behaviour:
During the phase in which i experience a replay i pulled a "state", with reward -100, a terminal state, in which the agent chose to run into a wall.
when i model.predict(state) I got: 39.8269 18.1576 21.3698 15.1639 The action in the memory is 2 with reward -100, so i changed to: 39.8269 18.1576 -100 15.1639
And let it train, i re-predicted and got: 40.0833 18.1578 21.1512 15.2231
Only a minor change for such a big error isn't it? Could this be the reason for my drop in performance? perhaps there is a different reason i'm missing?
Appendix: The game has a board of NxN the agent starts in the center (rounded up) Every other cell has a gold coin. When an agent steps into a cell with a gold coin he gets +10 and the coin is gone. When an agent steps into an empty cell he gets -1 reward. When an agent steps outside the array he gets reward -100 and the game ends. if the agent doesn't find gold in 6 turns the game ends and the agent gains -1.
The input layer is [0:3] are the last action performed in a one-hot-encoding  is the moves until starvation. [5:8] the square above the agent,below,right and left.[-1 wall, 1 empty cell, 2 gold coin] [9:12] are the squares in diagonal to the agent (UR,UL,DR,DL)