# A measure of redundancy in mutual information

Mutual information quantifies to what degree $$X$$ decreases the uncertainty about $$Y$$. However, to my understanding, it does not quantify "in how many ways" $$X$$ decreases the uncertainty. E.g., consider the case where $$X$$ is a 3D vector, and consider $$X_1=[Y,0,0]$$ vs. $$X_2 = [Y,Y^2, 3.5Y]$$. Intuitively, $$X_2$$ contains "more information" about $$Y$$, or is more redundant with respect to $$Y$$, than $$X_1$$; but if I understand correctly, both have the same mutual information. Is there an alternative information-theoretic measure that can quantify this difference?

Thanks!

# dit

This python package for discrete information theory provides a standard bivariate case based on

1. Basic Shannon measure of mutual information for bivariate distributions
2. Measures for multivariate distributions

Generalized as the smallest quantity that can be subtracted from the joint, and from each part of a partition of all the variables, such that the joint entropy minus this quantity is equal to the sum of each partition entropy minus this quantity.

local modification of a single variable can not increase the amount of correlation or dependence it has with the other variables.

# pyitlib

Library in python for information-theoretic methods.

Below are the mutual information measures found in pyitlib package

## Other measures in Research Communities

Apart from this, there are few best approaches as I have found in the research communities which looks promising to quantify measures of redundancy

1. Part mutual information: This new measure is based on information theory that accurately quantify nonlinearly direct associations between measured variables. For more information, part mutual information for quantifying direct associations
2. Calculate mutual information using recursive adaptive partitioning: This paper ideally focuses on mutual information between discrete variables with many categories using Recursive Adaptive Partitioning
3. Comparative redundancy calculations: A comparative study of existing redundancy calculations with new measure of bivariate redundancy measure. A Bivariate Measure of Redundant Information
4. Synergistic mutual information: briefly explains about how single PI-region is either redundant, unique or synergistic. Research paper: Quantifying synergistic mutual information
5. Partial Information Decomposition: a redundancy measure as proposed by Williams and Beer which typically introduce partial information atoms(PI-atoms) to decompose multivariate mutual information into non-negative terms. Refer to Nonnegative Decomposition of Multivariate Information
6. Absolute mutual information: This measure is calculated using algorithmic complexity
There are conditional versions of Kolmogorov complexity that could be applied to your problem. It will take more computation resources to specify $$X_1$$ relative to $$X_2$$ if both were given $$Y$$ as a prior.