5
$\begingroup$

I am looking to calculate the information contained in a neural network. I am also looking to calculate the maximum information contained by any neural network in a certain amount of bits. These two measures should be comparable (as in I can compare whether my current neural network has reached the max or is less than the max and by how much).

Information is relative, so I define it relative to the real a priori distribution of the data that the neural network is trying to estimate.

I have come across Von Neumann entropy which can be applied to a matrix, but because it is not additive I can't apply it to a series of weight matrices (assuming the weight matrices encode all the information of a neural network).

I found three other papers Entropy-Constrained raining of Deep Neural Networks , Entropy and mutual information in models of deep neural networks and Deep Learning and the Information Bottleneck Principle. The second contains a link to this github repo, but this method requires the activation functions and weight matrices to be known which is not the case for finding the max entropy of any neural network in n bits.

How can I calculate the amount of information contain in/entropy of a neural network? And how can I calculate the same measure for any neural network in n bits?

$\endgroup$
5
  • $\begingroup$ Welcome to DS SE! Currently, this post is a series of statements. Would you please update it by asking a question? $\endgroup$
    – Ben
    Jul 17, 2020 at 5:23
  • $\begingroup$ @BenjiAlbert done $\endgroup$
    – donkey
    Jul 17, 2020 at 14:16
  • $\begingroup$ Great (neat topic btw! +1). hopefully someone can answer you $\endgroup$
    – Ben
    Jul 17, 2020 at 16:41
  • 1
    $\begingroup$ My information theory is quite rusty. Is it actually possible to compute the entropy of a function (the neural network) relative to a distribution (the data)? $\endgroup$
    – zachdj
    Jul 22, 2020 at 20:38
  • $\begingroup$ @zachdj I don't think you can calculate the entropy of a function, but you can calculate the entropy of a distribution, so think about it instead as calculating the entropy of the posterior distribution of the data generated by that function maybe? I'm rusty on it as well. $\endgroup$
    – donkey
    Jul 23, 2020 at 17:56

1 Answer 1

1
$\begingroup$

To start, see Information Theory, Inference, and Learning Algorithms by David J.C. MacKay, starting with chapter 40 for information capacity of a single neuron (two bits per weight) through to at least chapter 42 for Hopfield Networks (fully connected feedback). The classic reference for information of a Hopfield Network is Information Capacity of the Hopfield Model by Abu-Mostafa & St. Jacques, but the textbook should have enough.

$\endgroup$
1
  • $\begingroup$ Thank you for the pointers. Is there a specific calculation you can cite that is most relevant to the question? $\endgroup$
    – donkey
    Jul 27, 2020 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.