6
$\begingroup$

I am trying to understand a neuroscience article:

  • Friston, Karl J., et al. "Action and behavior: a free-energy formulation." Biological cybernetics 102.3 (2010): 227-260. (DOI 10.1007/s00422-010-0364-z)

In this article, Friston gives three equations that are, as I understand him, equivalent or inter-convertertable and refer to both physical and Shannon entropy. They appear on page 231 of the article as equation (5):

The resulting expression for free-energy can be expressed in three ways (with the use of the Bayes rules and simple rearrangements):

• Energy minus entropy

• Divergence plus surprise

• Complexity minus accuracy

Mathematically, these correspond to:

\begin {aligned} F &= - \langle \ln p(\tilde s, \Psi \vert m) \rangle _q + \langle \ln q(\Psi \vert \mu) \rangle _q \ &= D(q(\Psi \vert \mu) \parallel p(\Psi \vert \tilde s, m)) - \ln p(\tilde s \vert m) & \qquad \qquad & (5) \ &= D(q(\Psi \vert \mu) \parallel p(\Psi \vert m)) - \langle \ln p(\tilde s \vert \Psi, m) \rangle _q \end{aligned}

The things I am struggling with at this point are:

  1. the meaning of the || in the 2nd and 3rd versions of the equations;
  2. and the negative logs.

Any help in understanding how these equations are actually what Fristen claims them to be would be greatly appreciated. For example, in the 1st equation, in what sense is the first term energy, etc?

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
3
$\begingroup$

I'm not qualified to understand almost all of that paper, but, I might be able to give some intuitions from information theory that help you parse the paper.

|| denotes the Kullback-Leibler divergence. It measures an information gain between two distributions. I suppose you could say it indicates the information in the real distribution of data that a model fails to capture.

When you see "negative log" think "entropy".

In the first equation, think of it as "-ln(...) - -ln(...)". This may help think of it as the difference of entropies. Likewise in the second, read it as "D(...) + -ln(...)". This may help think of it as "plus entropy".

If you look at the divergence definition, you'll see it is defined as the log of the ratio of the PDFs. This may help connect it to logs and negative logs. Look at the definition that writes it as cross-entropy minus entropy. Then this is all a question of differences of entropies of things which may be clearer.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks very much! Your comments are very helpful and provide a basis on which I may be able to tease more out of the article. John $\endgroup$
    – johnhidley
    Jul 14, 2014 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.