A measure of redundancy in mutual information

Question

Mutual information quantifies to what degree $X$ decreases the uncertainty about $Y$. However, to my understanding, it does not quantify "in how many ways" $X$ decreases the uncertainty. E.g., consider the case where $X$ is a 3D vector, and consider $X_1=[Y,0,0]$ vs. $X_2 = [Y,Y^2, 3.5Y]$. Intuitively, $X_2$ contains "more information" about $Y$, or is more redundant with respect to $Y$, than $X_1$; but if I understand correctly, both have the same mutual information. Is there an alternative information-theoretic measure that can quantify this difference?

Thanks!

Answer 1

Comparative study of Mutual Information measures

• A comparative study of standard Shannon entropy along with variations of Coding Theorem Method(CTM) and Block Decomposition Model(BTM) as shown in A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic Complexity
• General guide for Estimating mutual information and its applications in gene expression data and independent component analysis.

Information-theoretic Python Packages

dit

This python package for discrete information theory provides a standard bivariate case based on

1. Basic Shannon measure of mutual information for bivariate distributions
2. Measures for multivariate distributions
   • Co-Information: quantifies amount of information all variable participates in
   • Total Correlation: amount of information each individual variable carries above and beyond joint entropy
   • Dual Total Correlation: Also known as binding information is the amount of information shared among the variables.
   • Cohesion: spans total correlation to dual total correlation
   • CAEKL Mutual Information:

   Generalized as the smallest quantity that can be subtracted from the joint, and from each part of a partition of all the variables, such that the joint entropy minus this quantity is equal to the sum of each partition entropy minus this quantity.

   • Interaction Information: Equal in magnitude to co-information, however for odd number of variables takes the opposite sign
   • DeWeese-like Measures

   local modification of a single variable can not increase the amount of correlation or dependence it has with the other variables.

pyitlib

Library in python for information-theoretic methods.

Below are the mutual information measures found in pyitlib package

• Mutual information
• Normalised mutual information (7 variants)
• Variation of information
• Lautum information
• Conditional mutual information
• Co-information
• Interaction information
• Multi-information
• Binding information
• Residual entropy
• Exogenous local information
• Enigmatic information

Other measures in Research Communities

Apart from this, there are few best approaches as I have found in the research communities which looks promising to quantify measures of redundancy

1. Part mutual information: This new measure is based on information theory that accurately quantify nonlinearly direct associations between measured variables. For more information, part mutual information for quantifying direct associations
2. Calculate mutual information using recursive adaptive partitioning: This paper ideally focuses on mutual information between discrete variables with many categories using Recursive Adaptive Partitioning
3. Comparative redundancy calculations: A comparative study of existing redundancy calculations with new measure of bivariate redundancy measure. A Bivariate Measure of Redundant Information
4. Synergistic mutual information: briefly explains about how single PI-region is either redundant, unique or synergistic. Research paper: Quantifying synergistic mutual information
5. Partial Information Decomposition: a redundancy measure as proposed by Williams and Beer which typically introduce partial information atoms(PI-atoms) to decompose multivariate mutual information into non-negative terms. Refer to Nonnegative Decomposition of Multivariate Information
6. Absolute mutual information: This measure is calculated using algorithmic complexity
7. Pairwise adjusted mutual information
8. partial correlation: However it can only measure linear direct associations
9. Conditional mutual information quantify nonlinear direct relationships among variables, ideally for more than 2 variables

Answer 2

One way to quantify computational redundancy is with Kolmogorov complexity. Kolmogorov complexity is how complex a computer program is required to create an object.

There are conditional versions of Kolmogorov complexity that could be applied to your problem. It will take more computation resources to specify $X_1$ relative to $X_2$ if both were given $Y$ as a prior.