Questions tagged [information-theory]

Information theory is a branch of applied mathematics, electrical engineering, and computer science involving the quantification of information.

10 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2
votes
0answers
15 views

Given a sequence of inputs/outputs and a set of nodes that modify that input, can you find the topology of a graph?

I am working on a problem where I have to model a graph topology, where the nodes are logic/arithmetic operations that can be applied to the input. The network receives a multi-dimensional input, and ...
2
votes
0answers
74 views

Feature selection with information gain (KL divergence) and mutual information yields different results

I'm comparing different techniques for feature selection / feature ranking. Two of the techniques under scrutiny are the mutual information (MI) and the information gain (IG) as used in decision trees,...
2
votes
0answers
169 views

The meaning of the difference of two entropy values

I want to understanding the meaning of the difference of two information entropy values. I have the following scenario. Let $x$ be a number of hours a user spend on some video sharing websites. Thus, ...
1
vote
0answers
115 views

Visualizing mutual information of each convolution layer for image classification problem

I recently came across this paper where the author has proposed a compression based theory on understanding the layers of a DNN. In order to visualize what was going on the authors showed Figure 2 of ...
1
vote
0answers
69 views

How do you calculate the information capacity of a neural network?

Let's say I wanted to train a neural network to teach it the rules in a decision tree, so I generated a dataset by feeding arrays of random numbers into the pre-trained decision tree, and then used ...
1
vote
0answers
119 views

Information measure of rank changes?

I'm trying to compute the information content of a function that reranks a list. Perhaps more precisely, I'm trying to compare the information content of different arguments to that function. ...
1
vote
0answers
44 views

What does NIST information weights refer to?

NIST is a metric used to measure the goodness of translation. In the paper, Doddington (2002) introduce the notion of "Information weights" Information weights were computed using N-gram counts ...
0
votes
0answers
28 views

Can conditional entropy be used to derive an upper bound on predictive accuracy?

Say I've got two discrete random variables, X and Y. If I calculate 1 - H(Y|X), where ...
0
votes
0answers
14 views

Grey Information Theory

Grey System Theory was first published by Professor Deng in 1982. As far as I know, the theory encompasses and incorporates the notion of incomplete information into statistics. Statistical methods ...
0
votes
1answer
19 views

Entropy loss from collapsing/merging two categories

Suppose I am counting occurrences in a sequence. For a classical example, let's say I'm counting how many of each kind of car comes down a highway. After keeping tally for a while, I see there are ...