# Why use sampling instead of the mean value for policy in Reinforcement Learning?

I'm quite new in RL and I'm currently following David Silver's course on RL. But at the same time, I also want to get hands-on, so I followed this tutorial from Gymnasium documentation: https://gymnasium.farama.org/tutorials/training_agents/reinforce_invpend_gym_v26/

I understand the general concept and Idea, but I'm curious about why we should model the policy as a distribution (a Normal distribution in this case) and then take a sample from that distribution as an action to be applied to the RL environment.

Why don't we just use the mean value as an action instead of taking a sample from distribution as an action?

Here's the piece of code that I'm talking about:

def sample_action(self, state: np.ndarray) -> float:
"""Returns an action, conditioned on the policy and observation.

Args:
state: Observation from the environment

Returns:
action: Action to be performed
"""
state = torch.tensor(np.array([state]))
action_means, action_stddevs = self.net(state)

# create a normal distribution from the predicted
#   mean and standard deviation and sample an action
distrib = Normal(action_means[0] + self.eps, action_stddevs[0] + self.eps)
action = distrib.sample()
prob = distrib.log_prob(action)

action = action.numpy()

self.probs.append(prob)

return action


As an experiment, I have tried to change the action from action = distrib.sample() to action = action_means[0].

But it turns out that the model isn't learning.

Does anyone has an idea?

• Are you familiar with exploration in RL such as $\epsilon$-greedy soft policy for discrete action learning? Here for policy gradient method to deal with continuous action space it's similar via random action sampling otherwise agent won't learn the (approximately) optimal policy only through your proposed fixed mean value for each state/observation. Commented Mar 10, 2023 at 22:22