Mutual information quantifies to what degree $X$ decreases the uncertainty about $Y$. However, to my understanding, it does not quantify "in how many ways" $X$ decreases the uncertainty. E.g., consider the case where $X$ is a 3D vector, and consider $X_1=[Y,0,0]$ vs. $X_2 = [Y,Y^2, 3.5Y]$. Intuitively, $X_2$ contains "more information" about $Y$, or is more redundant with respect to $Y$, than $X_1$; but if I understand correctly, both have the same mutual information. Is there an alternative information-theoretic measure that can quantify this difference?



2 Answers 2


Comparative study of Mutual Information measures

Information-theoretic Python Packages


This python package for discrete information theory provides a standard bivariate case based on

  1. Basic Shannon measure of mutual information for bivariate distributions
  2. Measures for multivariate distributions

    Generalized as the smallest quantity that can be subtracted from the joint, and from each part of a partition of all the variables, such that the joint entropy minus this quantity is equal to the sum of each partition entropy minus this quantity.

    local modification of a single variable can not increase the amount of correlation or dependence it has with the other variables.


Library in python for information-theoretic methods.

Below are the mutual information measures found in pyitlib package

Other measures in Research Communities

Apart from this, there are few best approaches as I have found in the research communities which looks promising to quantify measures of redundancy

  1. Part mutual information: This new measure is based on information theory that accurately quantify nonlinearly direct associations between measured variables. For more information, part mutual information for quantifying direct associations
  2. Calculate mutual information using recursive adaptive partitioning: This paper ideally focuses on mutual information between discrete variables with many categories using Recursive Adaptive Partitioning
  3. Comparative redundancy calculations: A comparative study of existing redundancy calculations with new measure of bivariate redundancy measure. A Bivariate Measure of Redundant Information
  4. Synergistic mutual information: briefly explains about how single PI-region is either redundant, unique or synergistic. Research paper: Quantifying synergistic mutual information
  5. Partial Information Decomposition: a redundancy measure as proposed by Williams and Beer which typically introduce partial information atoms(PI-atoms) to decompose multivariate mutual information into non-negative terms. Refer to Nonnegative Decomposition of Multivariate Information
  6. Absolute mutual information: This measure is calculated using algorithmic complexity
  7. Pairwise adjusted mutual information
  8. partial correlation: However it can only measure linear direct associations
  9. Conditional mutual information quantify nonlinear direct relationships among variables, ideally for more than 2 variables

One way to quantify computational redundancy is with Kolmogorov complexity. Kolmogorov complexity is how complex a computer program is required to create an object.

There are conditional versions of Kolmogorov complexity that could be applied to your problem. It will take more computation resources to specify $X_1$ relative to $X_2$ if both were given $Y$ as a prior.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.