# RL Sutton book, initial estimate of q*(a) for 10 arm testbed

The Sutton book does not mention what the initial estimate is for q*(a) before the first reward is received. In this code repo that seems to go along with the book: Sutton code repo

They have initialized it with 0 per snippet below:

def __init__(self, kArm=10, epsilon=0., initial=0., stepSize=0.1, sampleAverages=False, UCBParam=None,


But the explanation for Figure 2.1 that shows the distribution of rewards for the 10 arms of the bandit says,

Figure 2.1: An example bandit problem from the 10-armed testbed. The true value q ⇤ (a) of each of the ten actions was selected according to a normal distribution with mean zero and unit variance, and then the actual rewards were selected according to a mean q ⇤ (a) unit variance normal distribution, as suggested by these gray distributions.

So should I initialize instead with np.random.randn()?

Edit: The distribution

Initialisation of your estimates is a different issue. If you know something about the distributions of the true action values, then it would make sense to use that. For instance you could set all action values to the mean expected true value. Which is $$0$$. However, you may also use $$0$$ if you have no idea about true values, as it is a simple arbitrary value.
• I guess setting up the true value of the q*(a) at the start of test and pulling all subsquent rewards from the N(q*(a), 1) distribution can be used to test exploration vs. exploitation but the graph posted above looks like results from a low number of trials. The code repo coments read, initialize real rewards with N(0,1) distribution and estimations with desired initial value (0). – mLstudent33 Aug 4 '19 at 11:55