Skip to main content

Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

Estimating conditional mutual information of discrete random variables with large domains

Given a dataset I want to estimate the conditional mutual information $I(X;Y|Z,W)$ for random variables $X, Y, Z, W$. The issue I'm facing is that the domains are rather large (say $\{0, \dots, 2^{12}\...
Georg Streich's user avatar
0 votes
0 answers
13 views

Hash function to decorrelate datasets

Consider three sets of data A, B and C. mutual information I(A:B)=I(A:C). Now I apply hashing function to A to obtain a new data A' with entropy H(A|B). does that imply I(A':C)=0.
Akash Nag's user avatar
1 vote
1 answer
70 views

Is it usual to obtain very different values of mutual information score using sci-kit learn?

Good day to everyone! I have a little question related to the function mutual_information_score of sci-kit learn. I've been using it to obtain the mutual information between two datasets A and B; ...
gengar123's user avatar
1 vote
0 answers
83 views

Model uncertainty quantification

I'm reading a paper about model uncertainty quantification. Specifically, it says epistemic uncertainty is a kind of uncertainty due to lack of knowledge about a particular region in the input space. ...
piero's user avatar
  • 61
0 votes
1 answer
43 views

Question about the definition of information bottleneck

I've read some interpreting "information bottleneck" as the loss of information of X regarding a random variable Y when a compressed random variable T is used. Could you please explain why ...
piplustwo's user avatar
  • 128
1 vote
1 answer
114 views

Do you have to use clustering with SciKit-Learn's Mutual Information metric?

I'd like to calculate the mutual information between two datasets, but I'd prefer not to cluster them first. I'm thinking of using SciKit-Learn's mutual_info_score ...
Connor's user avatar
  • 661
2 votes
2 answers
973 views

Correlation vs Mutual Information vs let-the-model-decide

I recently encountered the Mutual Information concept, and started reading on it. As I saw that it can get non-linear relations, it seemed to me that it might be a more powerful method to choose which ...
Álvaro V.'s user avatar
0 votes
1 answer
154 views

For feature selection, do we use Chi-squared with Mutual Information together?

Or do we only choose one out of two for categorical data.
O O's user avatar
  • 1
0 votes
1 answer
164 views

VIF Vs Mutual Info

I was searching for the best ways for feature selection in a regression problem & came across a post suggesting mutual info for regression, I tried the same on boston data set. The results were as ...
Rohan's user avatar
  • 3
1 vote
0 answers
24 views

clustering algorithms' evaluation [closed]

How can I show clustering performance of various clustering algorithms on various datasets using adjusted mutual information and adjusted rand index. for instance, the plot below .
Alireza's user avatar
  • 11
1 vote
1 answer
447 views

How to fix my CSV files? (ValueError: Found array with 0 sample(s) (shape=(0, 1)) while a minimum of 1 is required) [closed]

I have tried to import two csv files into df1 and df2. Concatenated them to make df3. I ...
Noob's user avatar
  • 113
0 votes
0 answers
489 views

Pipelines with categorical and nan values

I am trying a Regression model on a dataset which has categorical and numerical variables along with nan values. I want to use Pipelines for imputation and encoding purposes. Now I have a few ...
spectre's user avatar
  • 2,125
0 votes
2 answers
621 views

A measure of redundancy in mutual information

Mutual information quantifies to what degree $X$ decreases the uncertainty about $Y$. However, to my understanding, it does not quantify "in how many ways" $X$ decreases the uncertainty. E.g....
user1767774's user avatar
1 vote
1 answer
62 views

Understanding math notation in infoGAN paper

I'm reading this paper about mutual information in infoGAN infoGAN_paper_link and already have the code to run it. I pretty much found code for it which is fine and dandy except for the fact that I ...
zipline86's user avatar
  • 369
0 votes
1 answer
2k views

Mututal Information in sklearn

I expected sklearn's mutual_info_classif to give a value of 1 for the mutual information of a series of values with itself but instead I'm seeing results ranging between about 1.0 and 1.5. What am I ...
Matthew Walker's user avatar
1 vote
0 answers
21 views

How does Mutual Information handle background overlap

I have been reading about mutual information in Image Registration. It's in the literature that MI is better able to handle the cases with the large background where anatomical structures are not ...
pramesh's user avatar
  • 141
2 votes
1 answer
2k views

When should mutual information be used for feature selection over other feature selection methods like correlation, ANOVA , etc?

I have a data set with categorical and continuous/ordinal explanatory variables and continuous target variable. I tried to filter features using one-way ANOVA for categorical variables and using ...
Ankita Talwar's user avatar
3 votes
1 answer
53 views

Several independent variables based on the same underlying data

I got a data containing, among others, two feature variables, which are based from the same underlying data (i.e. have mutual information), but they convey different information/message. How to handle ...
user2340939's user avatar
1 vote
1 answer
21 views

Upper bound on 'relatedness'?

We have ~100 answers to a questionnaire with five questions (Q5). Independently from that, we have about 50, somewhat overlapping, features describing the people who answers the questions (F50). After ...
Kaare's user avatar
  • 267
3 votes
1 answer
316 views

Conditional Entropy and Mutual Information - Clustering evaluation

First of all, I am doing clustering and I have the true labels for my data. For evaluation, I am using the weighted average of the entropy values for each predicted cluster. I also came across with ...
ismlhk's user avatar
  • 33
3 votes
1 answer
987 views

Difference between Information Gain and Mutual Information for feature selection

What is the difference between information gain and mutual information? At this point, I understand that information gain is calculated between a random variable and target class for classification ...
Abhik Banerjee's user avatar
2 votes
1 answer
226 views

Feature selection with information gain (KL divergence) and mutual information yields different results

I'm comparing different techniques for feature selection / feature ranking. Two of the techniques under scrutiny are the mutual information (MI) and the information gain (IG) as used in decision trees,...
AutoMiner's user avatar
  • 169
2 votes
2 answers
982 views

Concept of Mutual Information

I want to get mutual information in iris dataset to select best feature but i confused about mutual information. What is concept of mutual information for selecting feature? Can anyone explain it in ...
Saeed's user avatar
  • 133
1 vote
1 answer
262 views

Visualizing mutual information of each convolution layer for image classification problem

I recently came across this paper where the author has proposed a compression based theory on understanding the layers of a DNN. In order to visualize what was going on the authors showed Figure 2 of ...
Ashesh's user avatar
  • 90
0 votes
1 answer
206 views

PMI between lemma vs surface

I was wondering whether it's possible to compute the some sort of pointwise mutual information between lemma and its surface form. First if we assume, ...
alvas's user avatar
  • 2,420
0 votes
1 answer
1k views

How to measure F1 score and NMI for clustering task?

I see the authors of this paper are measuring F1 and NMI scores to measure the clustering quality. However, I don't understand the algorithm of how they actually measure it. See the Evaluation Section....
Jay Patel's user avatar
6 votes
1 answer
2k views

How to estimate the mutual information numerically?

Suppose I have a sample {$z_i$}$_{i\in[0,N]}$ = {($x_i,y_i$)}$_{i\in[0,N]}$ which commes from a probability distribution $p_z(z)$. How can I use it to estimate the mutual information between X and Y ? ...
patapouf_ai's user avatar
1 vote
0 answers
161 views

Feature selection with linear interaction between variables and correlation with categorical response variable

I am searching for a feature selection algorithm able to select the minimum number (minimum redundancy) of relevant variables (maximum relevance) with respect to a categorical response variable. I ...
gc5's user avatar
  • 879
5 votes
3 answers
6k views

feature selection techniques

Is it always a good idea to remove features that have high mutual information with each other and to remove features that have very low mutual information with the target variable? Why or why not?
Armon Safai's user avatar