# Calculating the entropy of a neural network

I am looking to calculate the information contained in a neural network. I am also looking to calculate the maximum information contained by any neural network in a certain amount of bits. These two measures should be comparable (as in I can compare whether my current neural network has reached the max or is less than the max and by how much).

Information is relative, so I define it relative to the real a priori distribution of the data that the neural network is trying to estimate.

I have come across Von Neumann entropy which can be applied to a matrix, but because it is not additive I can't apply it to a series of weight matrices (assuming the weight matrices encode all the information of a neural network).

I found three other papers Entropy-Constrained raining of Deep Neural Networks , Entropy and mutual information in models of deep neural networks and Deep Learning and the Information Bottleneck Principle. The second contains a link to this github repo, but this method requires the activation functions and weight matrices to be known which is not the case for finding the max entropy of any neural network in n bits.

How can I calculate the amount of information contain in/entropy of a neural network? And how can I calculate the same measure for any neural network in n bits?

• Welcome to DS SE! Currently, this post is a series of statements. Would you please update it by asking a question?
– Ben
Jul 17, 2020 at 5:23
• @BenjiAlbert done Jul 17, 2020 at 14:16
• Great (neat topic btw! +1). hopefully someone can answer you
– Ben
Jul 17, 2020 at 16:41
• My information theory is quite rusty. Is it actually possible to compute the entropy of a function (the neural network) relative to a distribution (the data)? Jul 22, 2020 at 20:38
• @zachdj I don't think you can calculate the entropy of a function, but you can calculate the entropy of a distribution, so think about it instead as calculating the entropy of the posterior distribution of the data generated by that function maybe? I'm rusty on it as well. Jul 23, 2020 at 17:56