1
$\begingroup$

In the information theory, the entropy is a measure of uncertainty in some system. Being applied to agent policy, entropy shows how much the agent is uncertain about which action to make. In math notation, entropy of the policy is defined as : $$H(\pi) = -\sum \pi(a|s) \log \pi(a|s)$$ The value of entropy is always greater than zero and has a single maximum when the policy is uniform. In other words, all actions have the same probability. Entropy becomes minimal when our policy has 1 for some action and 0 for all others, which means that the agent is absolutely sure what to do. To prevent our agent from being stuck in the local minimum, we are subtracting the entropy from the loss function, punishing the agent for being too certain about the action to take.

The above excerpt is from Maxim Lapan in the book Deep Reinforcement Learning Hands-on page 254.

In code, it might look like :

 optimizer.zero_grad()
 logits= PG_network(batch_states_ts)
 log_prob = F.log_softmax(logits, dim=1)
 log_prob_actions = batch_scales_ts * log_prob[range(params["batch_size"]), batch_actions_ts]
 loss_policy = -log_prob_actions.mean()

 prob = F.softmax(logits, dim=1)
 entropy = -(prob * log_prob).sum(dim=1).mean()
 entropy_loss = params["entropy_beta"] * entropy
 loss = loss_policy - entropy_loss

I know that a disadvantage of using policy gradient is our agent can be stuck at a local minimum. Can you explain mathematically why subtracting the entropy from our policy will prevent our agent from being stuck in the local minimum ?

$\endgroup$
1
  • $\begingroup$ what he calls loss function, you call loss_policy. could you edit your question with the formula for this loss function? $\endgroup$
    – develarist
    Aug 28, 2020 at 22:55

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.