Why use sampling instead of the mean value for policy in Reinforcement Learning?

Question

I'm quite new in RL and I'm currently following David Silver's course on RL. But at the same time, I also want to get hands-on, so I followed this tutorial from Gymnasium documentation: https://gymnasium.farama.org/tutorials/training_agents/reinforce_invpend_gym_v26/

I understand the general concept and Idea, but I'm curious about why we should model the policy as a distribution (a Normal distribution in this case) and then take a sample from that distribution as an action to be applied to the RL environment.

Why don't we just use the mean value as an action instead of taking a sample from distribution as an action?

Here's the piece of code that I'm talking about:

def sample_action(self, state: np.ndarray) -> float:
        """Returns an action, conditioned on the policy and observation.

        Args:
            state: Observation from the environment

        Returns:
            action: Action to be performed
        """
        state = torch.tensor(np.array([state]))
        action_means, action_stddevs = self.net(state)

        # create a normal distribution from the predicted
        #   mean and standard deviation and sample an action
        distrib = Normal(action_means[0] + self.eps, action_stddevs[0] + self.eps)
        action = distrib.sample()
        prob = distrib.log_prob(action)

        action = action.numpy()

        self.probs.append(prob)

        return action

As an experiment, I have tried to change the action from action = distrib.sample() to action = action_means[0].

But it turns out that the model isn't learning.

Does anyone has an idea?

Answer 1

If the mean is used, the value is approximately the same over time. Thus the actions will be very similar over time, providing less opportunity for the model to learn what other actions could be useful (aka, the explore in exploit-explore concept).

If a sample is used, the value is random. The randomness is weighted by the value of previous actions. The current action exploits previously useful actions and explores also.