I'm quite new in RL and I'm currently following David Silver's course on RL. But at the same time, I also want to get hands-on, so I followed this tutorial from Gymnasium documentation: https://gymnasium.farama.org/tutorials/training_agents/reinforce_invpend_gym_v26/
I understand the general concept and Idea, but I'm curious about why we should model the policy as a distribution (a Normal distribution in this case) and then take a sample from that distribution as an action to be applied to the RL environment.
Why don't we just use the mean value as an action instead of taking a sample from distribution as an action?
Here's the piece of code that I'm talking about:
def sample_action(self, state: np.ndarray) -> float: """Returns an action, conditioned on the policy and observation. Args: state: Observation from the environment Returns: action: Action to be performed """ state = torch.tensor(np.array([state])) action_means, action_stddevs = self.net(state) # create a normal distribution from the predicted # mean and standard deviation and sample an action distrib = Normal(action_means + self.eps, action_stddevs + self.eps) action = distrib.sample() prob = distrib.log_prob(action) action = action.numpy() self.probs.append(prob) return action
As an experiment, I have tried to change the action from
action = distrib.sample() to
action = action_means.
But it turns out that the model isn't learning.
Does anyone has an idea?