# Questions tagged [information-theory]

Information theory is a branch of applied mathematics, electrical engineering, and computer science involving the quantification of information.

10 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
15 views

### Given a sequence of inputs/outputs and a set of nodes that modify that input, can you find the topology of a graph?

I am working on a problem where I have to model a graph topology, where the nodes are logic/arithmetic operations that can be applied to the input. The network receives a multi-dimensional input, and ...
74 views

### Feature selection with information gain (KL divergence) and mutual information yields different results

I'm comparing different techniques for feature selection / feature ranking. Two of the techniques under scrutiny are the mutual information (MI) and the information gain (IG) as used in decision trees,...
169 views

### The meaning of the difference of two entropy values

I want to understanding the meaning of the difference of two information entropy values. I have the following scenario. Let $x$ be a number of hours a user spend on some video sharing websites. Thus, ...
115 views

### Visualizing mutual information of each convolution layer for image classification problem

I recently came across this paper where the author has proposed a compression based theory on understanding the layers of a DNN. In order to visualize what was going on the authors showed Figure 2 of ...
69 views

### How do you calculate the information capacity of a neural network?

Let's say I wanted to train a neural network to teach it the rules in a decision tree, so I generated a dataset by feeding arrays of random numbers into the pre-trained decision tree, and then used ...
119 views

### Information measure of rank changes?

I'm trying to compute the information content of a function that reranks a list. Perhaps more precisely, I'm trying to compare the information content of different arguments to that function. ...
44 views

### What does NIST information weights refer to?

NIST is a metric used to measure the goodness of translation. In the paper, Doddington (2002) introduce the notion of "Information weights" Information weights were computed using N-gram counts ...
28 views

### Can conditional entropy be used to derive an upper bound on predictive accuracy?

Say I've got two discrete random variables, X and Y. If I calculate 1 - H(Y|X), where ...