Questions tagged [mutual-information]
mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.
27
questions
0
votes
0
answers
20
views
Question about the definition of information bottleneck
I've read some interpreting "information bottleneck" as the loss of information of X regarding a random variable Y when a compressed random variable T is used.
Could you please explain why ...
- 126
1
vote
1
answer
31
views
Do you have to use clustering with SciKit-Learn's Mutual Information metric?
I'd like to calculate the mutual information between two datasets, but I'd prefer not to cluster them first.
I'm thinking of using SciKit-Learn's mutual_info_score ...
- 557
0
votes
0
answers
13
views
how to use numpy mutual information correctly
i want to use principal component analysis-mutual information (PCA-MI) to have data representation from source which has source relevance of (value from smartinsole) and ouput variable (value from ...
- 31
2
votes
2
answers
181
views
Correlation vs Mutual Information vs let-the-model-decide
I recently encountered the Mutual Information concept, and started reading on it. As I saw that it can get non-linear relations, it seemed to me that it might be a more powerful method to choose which ...
- 25
0
votes
0
answers
53
views
MRMR score calculation for first predictor with package mRMRe
I am using the mRMRe package in R to calculate MRMR scores for features in a dataset.
The package documentation explains the calculation of the MRMR score as the following:
The scores method returns ...
0
votes
1
answer
93
views
For feature selection, do we use Chi-squared with Mutual Information together?
Or do we only choose one out of two for categorical data.
- 1
0
votes
1
answer
50
views
VIF Vs Mutual Info
I was searching for the best ways for feature selection in a regression problem & came across a post suggesting mutual info for regression, I tried the same on boston data set. The results were as ...
- 3
1
vote
0
answers
22
views
clustering algorithms' evaluation [closed]
How can I show clustering performance of various clustering algorithms on various datasets using adjusted mutual information and adjusted rand index.
for instance, the plot below
.
- 11
1
vote
1
answer
306
views
How to fix my CSV files? (ValueError: Found array with 0 sample(s) (shape=(0, 1)) while a minimum of 1 is required) [closed]
I have tried to import two csv files into df1 and df2. Concatenated them to make df3. I ...
- 113
0
votes
0
answers
407
views
Pipelines with categorical and nan values
I am trying a Regression model on a dataset which has categorical and numerical variables along with nan values. I want to use Pipelines for imputation and encoding purposes. Now I have a few ...
- 1,576
0
votes
2
answers
434
views
A measure of redundancy in mutual information
Mutual information quantifies to what degree $X$ decreases the uncertainty about $Y$. However, to my understanding, it does not quantify "in how many ways" $X$ decreases the uncertainty. E.g....
- 33
1
vote
1
answer
44
views
Understanding math notation in infoGAN paper
I'm reading this paper about mutual information in infoGAN infoGAN_paper_link and already have the code to run it. I pretty much found code for it which is fine and dandy except for the fact that I ...
- 349
0
votes
1
answer
1k
views
Mututal Information in sklearn
I expected sklearn's mutual_info_classif to give a value of 1 for the mutual information of a series of values with itself but instead I'm seeing results ranging between about 1.0 and 1.5. What am I ...
- 103
1
vote
0
answers
21
views
How does Mutual Information handle background overlap
I have been reading about mutual information in Image Registration. It's in the literature that MI is better able to handle the cases with the large background where anatomical structures are not ...
- 141
1
vote
1
answer
1k
views
When should mutual information be used for feature selection over other feature selection methods like correlation, ANOVA , etc?
I have a data set with categorical and continuous/ordinal explanatory variables and continuous target variable. I tried to filter features using one-way ANOVA for categorical variables and using ...
- 307
3
votes
1
answer
39
views
Several independent variables based on the same underlying data
I got a data containing, among others, two feature variables, which are based from the same underlying data (i.e. have mutual information), but they convey different information/message. How to handle ...
- 133
1
vote
1
answer
21
views
Upper bound on 'relatedness'?
We have ~100 answers to a questionnaire with five questions (Q5). Independently from that, we have about 50, somewhat overlapping, features describing the people who answers the questions (F50). After ...
- 267
3
votes
1
answer
221
views
Conditional Entropy and Mutual Information - Clustering evaluation
First of all, I am doing clustering and I have the true labels for my data.
For evaluation, I am using the weighted average of the entropy values for each predicted cluster. I also came across with ...
- 33
3
votes
1
answer
776
views
Difference between Information Gain and Mutual Information for feature selection
What is the difference between information gain and mutual information?
At this point, I understand that information gain is calculated between a random variable and target class for classification ...
- 141
2
votes
1
answer
191
views
Feature selection with information gain (KL divergence) and mutual information yields different results
I'm comparing different techniques for feature selection / feature ranking. Two of the techniques under scrutiny are the mutual information (MI) and the information gain (IG) as used in decision trees,...
- 169
2
votes
2
answers
917
views
Concept of Mutual Information
I want to get mutual information in iris dataset to select best feature but i confused about mutual information.
What is concept of mutual information for selecting feature?
Can anyone explain it in ...
- 133
1
vote
1
answer
218
views
Visualizing mutual information of each convolution layer for image classification problem
I recently came across this paper where the author has proposed a compression based theory on understanding the layers of a DNN. In order to visualize what was going on the authors showed Figure 2 of ...
- 90
0
votes
1
answer
197
views
PMI between lemma vs surface
I was wondering whether it's possible to compute the some sort of pointwise mutual information between lemma and its surface form.
First if we assume,
...
- 2,330
0
votes
1
answer
1k
views
How to measure F1 score and NMI for clustering task?
I see the authors of this paper are measuring F1 and NMI scores to measure the clustering quality. However, I don't understand the algorithm of how they actually measure it. See the Evaluation Section....
- 19
6
votes
1
answer
2k
views
How to estimate the mutual information numerically?
Suppose I have a sample {$z_i$}$_{i\in[0,N]}$ = {($x_i,y_i$)}$_{i\in[0,N]}$ which commes from a probability distribution $p_z(z)$. How can I use it to estimate the mutual information between X and Y ?
...
- 416
1
vote
0
answers
159
views
Feature selection with linear interaction between variables and correlation with categorical response variable
I am searching for a feature selection algorithm able to select the minimum number (minimum redundancy) of relevant variables (maximum relevance) with respect to a categorical response variable.
I ...
- 879
5
votes
3
answers
6k
views
feature selection techniques
Is it always a good idea to remove features that have high mutual information with each other and to remove features that have very low mutual information with the target variable? Why or why not?
- 419