Questions tagged [information-theory]
Information theory is a branch of applied mathematics, electrical engineering, and computer science involving the quantification of information.
36
questions
0
votes
0
answers
12
views
How to characterize conditional entropy of a variable given uniformly distributed variables
I'm trying to derive conclusions on how uniform distributions with different alphabet lengths influence the information measured from any other random variable (not neccesarily uniformly distributed). ...
- 101
0
votes
0
answers
20
views
Does the Lempel-Ziv-Welch algorithm have theoretical or practical use as a language model?
If we encode a string using the LZW algorithm, we obtain a dictionary which maps strings of increasing length onto output symbols and a sequence of output symbols.
Is the LZW algorithm useful (...
1
vote
0
answers
26
views
Shannon Information Content related to Uncertainty?
I'm a data scientist student currently writing my master thesis which resolves around the Cross Entropy (CE) Loss Function for neural networks. From my understanding, the CE is based on the Entropy, ...
- 11
1
vote
1
answer
26
views
How to calculate the information conveyed in a message for a given dataset
Given the data sets.
Test Set
...
- 151
0
votes
1
answer
47
views
What does "S" in Shannon's entropy stands for?
I see many machine learning texts using the following notation to represent Shannon's entropy in classification/supervised learning contexts:
$$
H(S) = \sum_{i \in Y}p_i \log(p_i)
$$
Where $p_i$ is ...
- 395
0
votes
2
answers
165
views
Self-Attention Summation and Loss of Information
In self-attention, the attention for a word is calculated as:
$$
A(q, K, V) = \sum_{i} \frac{exp(q.k^{<i>})}{\sum_{j} exp(q.k^{<j>})}v^{<i>}
$$
My question is why we sum over the ...
- 1
0
votes
2
answers
393
views
A measure of redundancy in mutual information
Mutual information quantifies to what degree $X$ decreases the uncertainty about $Y$. However, to my understanding, it does not quantify "in how many ways" $X$ decreases the uncertainty. E.g....
- 33
1
vote
1
answer
68
views
Information bottleneck and deep neural network
I learned about "the information bottleneck view of deep learning."
But in a nutshell, what does this tell us?
I don't see what the role is of depth in this approach as long as it is larger ...
- 153
5
votes
1
answer
575
views
Calculating the entropy of a neural network
I am looking to calculate the information contained in a neural network. I am also looking to calculate the maximum information contained by any neural network in a certain amount of bits. These two ...
- 101
2
votes
1
answer
83
views
Given a sequence of inputs/outputs and a set of nodes that modify that input, can you find the topology of a graph?
I am working on a problem where I have to model a graph topology, where the nodes are logic/arithmetic operations that can be applied to the input. The network receives a multi-dimensional input, and ...
- 121
1
vote
1
answer
34
views
How to measure the information of covariates in a ML task?
Background
Recently, I do 2 different ML projects.
One is lending club loan prediction, another is a pravite dataset in online experiment field to predict whether a customer will take the treatment....
- 111
0
votes
1
answer
191
views
Does Decision tree classifier calculate entropies before transforming categorical features using OneHotEncoder or transformation should be done
I am new to machine learning, and I've got to the point to drop out from it as online tutorials are pretty confusing as well.
Entropy and Decision trees
One of confusing tutorials was as the ...
- 163
1
vote
1
answer
31
views
What (probabilistic models) can only output decisions when they are certain?
I'm basically looking for approaches, models, algorithms for the following situation (a fault diagnosis problem):
I have an input set $\{x_i\}_{i \in \{1..m\}}$ with $n$ binary features of cases (...
- 11
2
votes
1
answer
196
views
Conditional Entropy and Mutual Information - Clustering evaluation
First of all, I am doing clustering and I have the true labels for my data.
For evaluation, I am using the weighted average of the entropy values for each predicted cluster. I also came across with ...
- 23
1
vote
1
answer
706
views
Choosing the first node in a decision tree, basic example
I'm wondering whether I'm understanding the process of choosing a node correctly and would like to see if this example makes sense.
using the following data :
...
- 173
2
votes
1
answer
182
views
Feature selection with information gain (KL divergence) and mutual information yields different results
I'm comparing different techniques for feature selection / feature ranking. Two of the techniques under scrutiny are the mutual information (MI) and the information gain (IG) as used in decision trees,...
- 169
0
votes
1
answer
37
views
Entropy loss from collapsing/merging two categories
Suppose I am counting occurrences in a sequence. For a classical example, let's say I'm counting how many of each kind of car comes down a highway.
After keeping tally for a while, I see there are ...
- 74
1
vote
0
answers
33
views
How to measure word similarity using wordnet for the information theoretic definition as detailed in Resnik 1995? [closed]
Resnik 1995 equation 3 uses count(n) to define P(c). What is count(n)?
Any solved example would be appreciated.
- 111
1
vote
1
answer
209
views
Visualizing mutual information of each convolution layer for image classification problem
I recently came across this paper where the author has proposed a compression based theory on understanding the layers of a DNN. In order to visualize what was going on the authors showed Figure 2 of ...
- 90
3
votes
1
answer
78
views
Sparse IR with user feedback
I'm considering a problem framing within an information retrieval context.
I have a sequence of documents that feature different attributes. In the web context, these would be webpages. One ...
- 83
6
votes
1
answer
2k
views
Is it valid to include your validation data in your vocabulary for NLP?
At the moment, I am following best practices and creating a "bag of words" vector with a vocabulary from the training data. My cross validation (and test) datasets are transformed using this model, ...
- 163
7
votes
1
answer
236
views
Are deep learning models way over the required capacity for their datasets' estimated entropies?
this question might seem a bit odd. I was doing some self-studies into information theory and decided to do some more formal investigations into deep learning. Please bear with me as I try to explain.
...
1
vote
0
answers
105
views
How do you calculate the information capacity of a neural network?
Let's say I wanted to train a neural network to teach it the rules in a decision tree, so I generated a dataset by feeding arrays of random numbers into the pre-trained decision tree, and then used ...
2
votes
1
answer
222
views
In CS231n lecture, can't the linear classifier be softmax itself?
I am little bit confused on why the scoring function that is the $f(X,W)$ is chosen to be $W,X$ while they talk about Softmax and SVM loss in this.
Can't they take Softmax classifier or SVM ...
2
votes
1
answer
71
views
After a Feature Scaling do i have the same initial information?
I'm studying the gradient descent algorithm for single hidden layer neural networks.
Suppose that I have an initial dataset and then I use mean normalization in order to scale the features.
Why ...
- 451
-1
votes
1
answer
280
views
Measure information gain / loss after a data Transformation operation [closed]
Suppose that we have a dataset of 2 samples :
[{1,2,0}, {2,0,0}, {3,1,1}, {4,0,1}, {5,1,1}]
(the last element of each row is the class variable)
If we want to ...
- 119
1
vote
0
answers
313
views
Information measure of rank changes?
I'm trying to compute the information content of a function that reranks a list. Perhaps more precisely, I'm trying to compare the information content of different arguments to that function.
...
- 181
1
vote
0
answers
87
views
What does NIST information weights refer to?
NIST is a metric used to measure the goodness of translation.
In the paper, Doddington (2002) introduce the notion of "Information weights"
Information weights were computed using N-gram ...
- 2,482
2
votes
2
answers
3k
views
Sigmoid's stability
Analytically, the logarithm of the sigmoid is always defined and
finite, because the sigmoid returns values restricted to the open
interval (0, 1), rather than using the entire closed interval of ...
- 131
0
votes
1
answer
2k
views
Softmax cost function
In this post, https://cs231n.github.io/linear-classify/#webdemo it mentions "Softmax classifier interprets the scores inside the output vector f as the unnormalized log probabilities." Aren't we ...
- 131
0
votes
1
answer
4k
views
A practical reason to use Cross-entropy as a error-function in Neural networks?
Cross-entropy tends to allow errors to change weights even when nodes
saturate (which means that their derivatives are asymptotically close
to 0.)
Link
Why is the above statement true? Figures ...
- 131
3
votes
0
answers
382
views
The meaning of the difference of two entropy values
I want to understanding the meaning of the difference of two information entropy values.
I have the following scenario. Let $x$ be a number of hours a user spend on some video sharing websites. Thus, ...
- 181
4
votes
3
answers
2k
views
Why the number of neurons or convolutions chosen equal powers of two?
In the overwhelming number of works devoted to the neural networks, the authors suggest arhitechure in which each layer is a numbers of neurons is power of 2
what are the theoretical reasons(...
- 143
6
votes
1
answer
8k
views
Why we use information gain over accuracy as splitting criterion in decision tree?
In decision tree classifier most of the algorithms use Information gain as spiting criterion. We select the feature with maximum information gain to split on.
I think that using accuracy instead of ...
- 257
5
votes
1
answer
2k
views
How to estimate the mutual information numerically?
Suppose I have a sample {$z_i$}$_{i\in[0,N]}$ = {($x_i,y_i$)}$_{i\in[0,N]}$ which commes from a probability distribution $p_z(z)$. How can I use it to estimate the mutual information between X and Y ?
...
- 406
108
votes
9
answers
133k
views
When should I use Gini Impurity as opposed to Information Gain (Entropy)?
Can someone practically explain the rationale behind Gini impurity vs Information gain (based on Entropy)?
Which metric is better to use in different scenarios while using decision trees?
- 1,181