0
$\begingroup$

I've built a game with an agent moving around trying to collect gold (+10 reward) and it dies when it hits a wall (-100, terminal condition) and -1 for any other step that is not gold nor wall. My action space is 4 (U, D,R,L) and my state is the sight he has. [More details in the end of the message]

The problem is it improves for a few hundred\thousands (depending on the model) iterations and then it drops to into a mode it simply runs into a wall, never improving from there.

This is my model:

            self.model = K.models.Sequential()
        self.model.add(K.layers.Dense(128, input_shape=(self.observation_space,), activation="relu"))
        self.model.add(K.layers.Dropout(0.1))
        self.model.add(K.layers.Dense(64, activation="relu"))
        self.model.add(K.layers.Dropout(0.1))
        self.model.add(K.layers.Dense(self.action_space, activation="linear"))
        optimizer = K.optimizers.Adam(lr=0.01)
        self.model.compile(loss="mse", optimizer=optimizer)

I've tried debugging it, and encountered the following behaviour:

During the phase in which i experience a replay i pulled a "state", with reward -100, a terminal state, in which the agent chose to run into a wall.

when i model.predict(state) I got: 39.8269 18.1576 21.3698 15.1639 The action in the memory is 2 with reward -100, so i changed to: 39.8269 18.1576 -100 15.1639

And let it train, i re-predicted and got: 40.0833 18.1578 21.1512 15.2231

Only a minor change for such a big error isn't it? Could this be the reason for my drop in performance? perhaps there is a different reason i'm missing?

Appendix: The game has a board of NxN the agent starts in the center (rounded up) Every other cell has a gold coin. When an agent steps into a cell with a gold coin he gets +10 and the coin is gone. When an agent steps into an empty cell he gets -1 reward. When an agent steps outside the array he gets reward -100 and the game ends. if the agent doesn't find gold in 6 turns the game ends and the agent gains -1.

The input layer is [0:3] are the last action performed in a one-hot-encoding [5] is the moves until starvation. [5:8] the square above the agent,below,right and left.[-1 wall, 1 empty cell, 2 gold coin] [9:12] are the squares in diagonal to the agent (UR,UL,DR,DL)

Thank you!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.