I've built a game with an agent moving around trying to collect gold (+10 reward) and it dies when it hits a wall (-100, terminal condition) and -1 for any other step that is not gold nor wall. My action space is 4 (U, D,R,L) and my state is the sight he has. [More details in the end of the message]

The problem is it improves for a few hundred\thousands (depending on the model) iterations and then it drops to into a mode it simply runs into a wall, never improving from there.

This is my model:

            self.model = K.models.Sequential()
        self.model.add(K.layers.Dense(128, input_shape=(self.observation_space,), activation="relu"))
        self.model.add(K.layers.Dense(64, activation="relu"))
        self.model.add(K.layers.Dense(self.action_space, activation="linear"))
        optimizer = K.optimizers.Adam(lr=0.01)
        self.model.compile(loss="mse", optimizer=optimizer)

I've tried debugging it, and encountered the following behaviour:

During the phase in which i experience a replay i pulled a "state", with reward -100, a terminal state, in which the agent chose to run into a wall.

when i model.predict(state) I got: 39.8269 18.1576 21.3698 15.1639 The action in the memory is 2 with reward -100, so i changed to: 39.8269 18.1576 -100 15.1639

And let it train, i re-predicted and got: 40.0833 18.1578 21.1512 15.2231

Only a minor change for such a big error isn't it? Could this be the reason for my drop in performance? perhaps there is a different reason i'm missing?

Appendix: The game has a board of NxN the agent starts in the center (rounded up) Every other cell has a gold coin. When an agent steps into a cell with a gold coin he gets +10 and the coin is gone. When an agent steps into an empty cell he gets -1 reward. When an agent steps outside the array he gets reward -100 and the game ends. if the agent doesn't find gold in 6 turns the game ends and the agent gains -1.

The input layer is [0:3] are the last action performed in a one-hot-encoding [5] is the moves until starvation. [5:8] the square above the agent,below,right and left.[-1 wall, 1 empty cell, 2 gold coin] [9:12] are the squares in diagonal to the agent (UR,UL,DR,DL)

Thank you!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.