Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

Filter by
Sorted by
Tagged with
0
votes
1answer
26 views

When should mutual information be used for feature selection over other feature selection methods like correlation, ANOVA , etc?

I have a data set with categorical and continuous/ordinal explanatory variables and continuous target variable. I tried to filter features using one-way ANOVA for categorical variables and using ...
2
votes
1answer
26 views

Several independent variables based on the same underlying data

I got a data containing, among others, two feature variables, which are based from the same underlying data (i.e. have mutual information), but they convey different information/message. How to handle ...
1
vote
1answer
16 views

Upper bound on 'relatedness'?

We have ~100 answers to a questionnaire with five questions (Q5). Independently from that, we have about 50, somewhat overlapping, features describing the people who answers the questions (F50). After ...
0
votes
0answers
28 views

Can conditional entropy be used to derive an upper bound on predictive accuracy?

Say I've got two discrete random variables, X and Y. If I calculate 1 - H(Y|X), where ...
2
votes
1answer
60 views

Conditional Entropy and Mutual Information - Clustering evaluation

First of all, I am doing clustering and I have the true labels for my data. For evaluation, I am using the weighted average of the entropy values for each predicted cluster. I also came across with ...
2
votes
1answer
209 views

Difference between Information Gain and Mutual Information for feature selection

What is the difference between information gain and mutual information? At this point, I understand that information gain is calculated between a random variable and target class for classification ...
2
votes
0answers
77 views

Feature selection with information gain (KL divergence) and mutual information yields different results

I'm comparing different techniques for feature selection / feature ranking. Two of the techniques under scrutiny are the mutual information (MI) and the information gain (IG) as used in decision trees,...
2
votes
2answers
386 views

Concept of Mutual Information

I want to get mutual information in iris dataset to select best feature but i confused about mutual information. What is concept of mutual information for selecting feature? Can anyone explain it in ...
1
vote
0answers
117 views

Visualizing mutual information of each convolution layer for image classification problem

I recently came across this paper where the author has proposed a compression based theory on understanding the layers of a DNN. In order to visualize what was going on the authors showed Figure 2 of ...
0
votes
1answer
146 views

PMI between lemma vs surface

I was wondering whether it's possible to compute the some sort of pointwise mutual information between lemma and its surface form. First if we assume, ...
0
votes
1answer
987 views

How to measure F1 score and NMI for clustering task?

I see the authors of this paper are measuring F1 and NMI scores to measure the clustering quality. However, I don't understand the algorithm of how they actually measure it. See the Evaluation Section....
1
vote
0answers
130 views

Modeling data combination from multiple sources

I am currently working on my masters thesis and have come to the point where I need to combine data from different sources to find new patterns. An example could be the following: I have transactions ...
3
votes
1answer
1k views

How to estimate the mutual information numerically?

Suppose I have a sample {$z_i$}$_{i\in[0,N]}$ = {($x_i,y_i$)}$_{i\in[0,N]}$ which commes from a probability distribution $p_z(z)$. How can I use it to estimate the mutual information between X and Y ? ...
1
vote
0answers
149 views

Feature selection with linear interaction between variables and correlation with categorical response variable

I am searching for a feature selection algorithm able to select the minimum number (minimum redundancy) of relevant variables (maximum relevance) with respect to a categorical response variable. I ...
5
votes
3answers
5k views

feature selection techniques

Is it always a good idea to remove features that have high mutual information with each other and to remove features that have very low mutual information with the target variable? Why or why not?