# Implementing experience replay in reinforcement learning

I've been reading Google's DeepMind Atari paper and I'm trying to understand how to implement experience replay. My question is that whether we update the parameters $\theta$ of function $Q$ once for all the samples of the minibatch or we do that for each sample of the minibatch separately?

According to the following code from this paper, it performs the gradient descent on loss term for the $j$-th sample. However, I have seen other papers (referring to this paper) that say that we first calculate the sum of loss terms for all samples of the minibatch and then perform the gradient descent on this sum of losses.

• Are you asking if to use stochastic gradient descent (SGD), i.e. update the parameters of the network for each item in the sample, or GD (i.e. perform an update of the parameters of the network using the whole sample)? I think you can do both. It depends. If you use SGD, you will have to update the parameters multiple times. If you use SG, you will to update the parameters only once. But the losses will be slightly different. Have you tried to look at the existing implementations? – nbro Feb 14 at 15:58