# Is reward accumulated during a play iteration when performing SARSA?

I've been having issue with getting my DQN to converge to a good solution for snake. Regardless of the different types of reward functions I've tried, it seems that the snake is indefinitely going around in circles. I have not tried exploring more states yet because I am confused about how to properly assign reward.

Currently, I am using a 2D-Gaussian distribution to assign reward where $$f(x=x_{food},y=y_{food}) = 1$$. Terminal states like hitting the wall or itself result in a reward value of -1.

My reason for using the Gaussian was because of the relatively sparse rewards in this game and the ability to easily clip rewards between [1,-1] in meaning full way.

I have two questions.

1. Is this an appropriate way to define the reward function?
2. Currently during training, I do no accumulate the reward during each play iteration. So each reward for transitions are independent of the reward values before it. Right now I am doing $$\big[S_1,A_1,R_1,S_2,A_2,R_2,S_3\big]$$. I've looked at other code where people have accumulate the reward like $$\big[S_1,A_1,R_1,S_2,A_2,R_2 = R_1+r,S_3\big]$$. Where $$r$$ is given by the reward function. The thing is, I can't find a paper that defines why you should do this. So my question is, which way is the appropriate way to assign reward?