# Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

27 questions
Filter by
Sorted by
Tagged with
2k views

### How to estimate the mutual information numerically?

Suppose I have a sample {$z_i$}$_{i\in[0,N]}$ = {($x_i,y_i$)}$_{i\in[0,N]}$ which commes from a probability distribution $p_z(z)$. How can I use it to estimate the mutual information between X and Y ? ...
• 416
6k views

### feature selection techniques

Is it always a good idea to remove features that have high mutual information with each other and to remove features that have very low mutual information with the target variable? Why or why not?
• 419
48 views

### Several independent variables based on the same underlying data

I got a data containing, among others, two feature variables, which are based from the same underlying data (i.e. have mutual information), but they convey different information/message. How to handle ...
• 133
946 views

### Difference between Information Gain and Mutual Information for feature selection

What is the difference between information gain and mutual information? At this point, I understand that information gain is calculated between a random variable and target class for classification ...
300 views

### Conditional Entropy and Mutual Information - Clustering evaluation

First of all, I am doing clustering and I have the true labels for my data. For evaluation, I am using the weighted average of the entropy values for each predicted cluster. I also came across with ...
• 33
966 views

### Concept of Mutual Information

I want to get mutual information in iris dataset to select best feature but i confused about mutual information. What is concept of mutual information for selecting feature? Can anyone explain it in ...
• 133
688 views

### Correlation vs Mutual Information vs let-the-model-decide

I recently encountered the Mutual Information concept, and started reading on it. As I saw that it can get non-linear relations, it seemed to me that it might be a more powerful method to choose which ...
1k views

### When should mutual information be used for feature selection over other feature selection methods like correlation, ANOVA , etc?

I have a data set with categorical and continuous/ordinal explanatory variables and continuous target variable. I tried to filter features using one-way ANOVA for categorical variables and using ...
222 views

### Feature selection with information gain (KL divergence) and mutual information yields different results

I'm comparing different techniques for feature selection / feature ranking. Two of the techniques under scrutiny are the mutual information (MI) and the information gain (IG) as used in decision trees,...
• 169
1 vote
91 views

### Do you have to use clustering with SciKit-Learn's Mutual Information metric?

I'd like to calculate the mutual information between two datasets, but I'd prefer not to cluster them first. I'm thinking of using SciKit-Learn's mutual_info_score ...
• 631
1 vote
416 views

### How to fix my CSV files? (ValueError: Found array with 0 sample(s) (shape=(0, 1)) while a minimum of 1 is required) [closed]

I have tried to import two csv files into df1 and df2. Concatenated them to make df3. I ...
• 113
1 vote
57 views

### Understanding math notation in infoGAN paper

I'm reading this paper about mutual information in infoGAN infoGAN_paper_link and already have the code to run it. I pretty much found code for it which is fine and dandy except for the fact that I ...
• 359
1 vote
21 views

### Upper bound on 'relatedness'?

We have ~100 answers to a questionnaire with five questions (Q5). Independently from that, we have about 50, somewhat overlapping, features describing the people who answers the questions (F50). After ...
• 267
1 vote
257 views

### Visualizing mutual information of each convolution layer for image classification problem

I recently came across this paper where the author has proposed a compression based theory on understanding the layers of a DNN. In order to visualize what was going on the authors showed Figure 2 of ...
• 90
1 vote
80 views

### Model uncertainty quantification

I'm reading a paper about model uncertainty quantification. Specifically, it says epistemic uncertainty is a kind of uncertainty due to lack of knowledge about a particular region in the input space. ...
• 61
1 vote
24 views

### clustering algorithms' evaluation [closed]

How can I show clustering performance of various clustering algorithms on various datasets using adjusted mutual information and adjusted rand index. for instance, the plot below .
• 11
1 vote
21 views

### How does Mutual Information handle background overlap

I have been reading about mutual information in Image Registration. It's in the literature that MI is better able to handle the cases with the large background where anatomical structures are not ...
• 141
1 vote
161 views

### Feature selection with linear interaction between variables and correlation with categorical response variable

I am searching for a feature selection algorithm able to select the minimum number (minimum redundancy) of relevant variables (maximum relevance) with respect to a categorical response variable. I ...
• 879
205 views

### PMI between lemma vs surface

I was wondering whether it's possible to compute the some sort of pointwise mutual information between lemma and its surface form. First if we assume, ...
• 2,410
568 views

### A measure of redundancy in mutual information

Mutual information quantifies to what degree $X$ decreases the uncertainty about $Y$. However, to my understanding, it does not quantify "in how many ways" $X$ decreases the uncertainty. E.g....
41 views

### Question about the definition of information bottleneck

I've read some interpreting "information bottleneck" as the loss of information of X regarding a random variable Y when a compressed random variable T is used. Could you please explain why ...
• 128
2k views

### Mututal Information in sklearn

I expected sklearn's mutual_info_classif to give a value of 1 for the mutual information of a series of values with itself but instead I'm seeing results ranging between about 1.0 and 1.5. What am I ...
31 views

### Is it usual to obtain very different values of mutual information score using sci-kit learn?

Good day to everyone! I have a little question related to the function mutual_information_score of sci-kit learn. I've been using it to obtain the mutual information between two datasets A and B; ...
135 views

### For feature selection, do we use Chi-squared with Mutual Information together?

Or do we only choose one out of two for categorical data.
• 1
130 views

### VIF Vs Mutual Info

I was searching for the best ways for feature selection in a regression problem & came across a post suggesting mutual info for regression, I tried the same on boston data set. The results were as ...