I'm quite new in RL and I'm currently following David Silver's course on RL. But at the same time, I also want to get hands-on, so I followed this tutorial from Gymnasium documentation: https://gymnasium.farama.org/tutorials/training_agents/reinforce_invpend_gym_v26/

I understand the general concept and Idea, but I'm curious about why we should model the policy as a distribution (a Normal distribution in this case) and then take a sample from that distribution as an action to be applied to the RL environment.

Why don't we just use the mean value as an action instead of taking a sample from distribution as an action?

Here's the piece of code that I'm talking about:

def sample_action(self, state: np.ndarray) -> float:
        """Returns an action, conditioned on the policy and observation.

            state: Observation from the environment

            action: Action to be performed
        state = torch.tensor(np.array([state]))
        action_means, action_stddevs = self.net(state)

        # create a normal distribution from the predicted
        #   mean and standard deviation and sample an action
        distrib = Normal(action_means[0] + self.eps, action_stddevs[0] + self.eps)
        action = distrib.sample()
        prob = distrib.log_prob(action)

        action = action.numpy()


        return action

As an experiment, I have tried to change the action from action = distrib.sample() to action = action_means[0].

But it turns out that the model isn't learning.

Does anyone has an idea?

  • $\begingroup$ Are you familiar with exploration in RL such as $\epsilon$-greedy soft policy for discrete action learning? Here for policy gradient method to deal with continuous action space it's similar via random action sampling otherwise agent won't learn the (approximately) optimal policy only through your proposed fixed mean value for each state/observation. $\endgroup$
    – cinch
    Commented Mar 10, 2023 at 22:22

1 Answer 1


If the mean is used, the value is approximately the same over time. Thus the actions will be very similar over time, providing less opportunity for the model to learn what other actions could be useful (aka, the explore in exploit-explore concept).

If a sample is used, the value is random. The randomness is weighted by the value of previous actions. The current action exploits previously useful actions and explores also.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.