1
$\begingroup$

I've been having issue with getting my DQN to converge to a good solution for snake. Regardless of the different types of reward functions I've tried, it seems that the snake is indefinitely going around in circles. I have not tried exploring more states yet because I am confused about how to properly assign reward.

Currently, I am using a 2D-Gaussian distribution to assign reward where $f(x=x_{food},y=y_{food}) = 1$. Terminal states like hitting the wall or itself result in a reward value of -1.

My reason for using the Gaussian was because of the relatively sparse rewards in this game and the ability to easily clip rewards between [1,-1] in meaning full way.

I have two questions.

  1. Is this an appropriate way to define the reward function?
  2. Currently during training, I do no accumulate the reward during each play iteration. So each reward for transitions are independent of the reward values before it. Right now I am doing $\big[S_1,A_1,R_1,S_2,A_2,R_2,S_3\big]$. I've looked at other code where people have accumulate the reward like $\big[S_1,A_1,R_1,S_2,A_2,R_2 = R_1+r,S_3\big]$. Where $r$ is given by the reward function. The thing is, I can't find a paper that defines why you should do this. So my question is, which way is the appropriate way to assign reward?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.