Questions tagged [information-theory]

Information theory is a branch of applied mathematics, electrical engineering, and computer science involving the quantification of information.

Filter by
Sorted by
Tagged with
0
votes
1answer
25 views

Does Decision tree classifier calculate entropies before transforming categorical features using OneHotEncoder or transformation should be done

I am new to machine learning, and I've got to the point to drop out from it as online tutorials are pretty confusing as well. Entropy and Decision trees One of confusing tutorials was as the ...
1
vote
1answer
28 views

What (probabilistic models) can only output decisions when they are certain?

I'm basically looking for approaches, models, algorithms for the following situation (a fault diagnosis problem): I have an input set $\{x_i\}_{i \in \{1..m\}}$ with $n$ binary features of cases (...
0
votes
0answers
22 views

Can conditional entropy be used to derive an upper bound on predictive accuracy?

Say I've got two discrete random variables, X and Y. If I calculate 1 - H(Y|X), where ...
0
votes
0answers
12 views

Grey Information Theory

Grey System Theory was first published by Professor Deng in 1982. As far as I know, the theory encompasses and incorporates the notion of incomplete information into statistics. Statistical methods ...
2
votes
1answer
41 views

Conditional Entropy and Mutual Information - Clustering evaluation

First of all, I am doing clustering and I have the true labels for my data. For evaluation, I am using the weighted average of the entropy values for each predicted cluster. I also came across with ...
1
vote
1answer
32 views

Choosing the first node in a decision tree, basic example

I'm wondering whether I'm understanding the process of choosing a node correctly and would like to see if this example makes sense. using the following data : ...
1
vote
0answers
55 views

Feature selection with information gain (KL divergence) and mutual information yields different results

I'm comparing different techniques for feature selection / feature ranking. Two of the techniques under scrutiny are the mutual information (MI) and the information gain (IG) as used in decision trees,...
0
votes
1answer
18 views

Entropy loss from collapsing/merging two categories

Suppose I am counting occurrences in a sequence. For a classical example, let's say I'm counting how many of each kind of car comes down a highway. After keeping tally for a while, I see there are ...
1
vote
0answers
23 views

How to measure word similarity using wordnet for the information theoretic definition as detailed in Resnik 1995? [closed]

Resnik 1995 equation 3 uses count(n) to define P(c). What is count(n)? Any solved example would be appreciated.
1
vote
0answers
101 views

Visualizing mutual information of each convolution layer for image classification problem

I recently came across this paper where the author has proposed a compression based theory on understanding the layers of a DNN. In order to visualize what was going on the authors showed Figure 2 of ...
3
votes
1answer
74 views

Sparse IR with user feedback

I'm considering a problem framing within an information retrieval context. I have a sequence of documents that feature different attributes. In the web context, these would be webpages. One ...
5
votes
1answer
524 views

Is it valid to include your validation data in your vocabulary for NLP?

At the moment, I am following best practices and creating a "bag of words" vector with a vocabulary from the training data. My cross validation (and test) datasets are transformed using this model, ...
7
votes
1answer
148 views

Are deep learning models way over the required capacity for their datasets' estimated entropies?

this question might seem a bit odd. I was doing some self-studies into information theory and decided to do some more formal investigations into deep learning. Please bear with me as I try to explain. ...
1
vote
0answers
65 views

How do you calculate the information capacity of a neural network?

Let's say I wanted to train a neural network to teach it the rules in a decision tree, so I generated a dataset by feeding arrays of random numbers into the pre-trained decision tree, and then used ...
2
votes
1answer
187 views

In CS231n lecture, can't the linear classifier be softmax itself?

I am little bit confused on why the scoring function that is the $f(X,W)$ is chosen to be $W,X$ while they talk about Softmax and SVM loss in this. Can't they take Softmax classifier or SVM ...
2
votes
1answer
66 views

After a Feature Scaling do i have the same initial information?

I'm studying the gradient descent algorithm for single hidden layer neural networks. Suppose that I have an initial dataset and then I use mean normalization in order to scale the features. Why ...
-1
votes
1answer
123 views

Measure information gain / loss after a data Transformation operation [closed]

Suppose that we have a dataset of 2 samples : [{1,2,0}, {2,0,0}, {3,1,1}, {4,0,1}, {5,1,1}] (the last element of each row is the class variable) If we want to ...
1
vote
0answers
87 views

Information measure of rank changes?

I'm trying to compute the information content of a function that reranks a list. Perhaps more precisely, I'm trying to compare the information content of different arguments to that function. ...
1
vote
0answers
37 views

What does NIST information weights refer to?

NIST is a metric used to measure the goodness of translation. In the paper, Doddington (2002) introduce the notion of "Information weights" Information weights were computed using N-gram counts ...
1
vote
2answers
2k views

Sigmoid's stability

Analytically, the logarithm of the sigmoid is always defined and finite, because the sigmoid returns values restricted to the open interval (0, 1), rather than using the entire closed interval of ...
0
votes
1answer
1k views

Softmax cost function

In this post, https://cs231n.github.io/linear-classify/#webdemo it mentions "Softmax classifier interprets the scores inside the output vector f as the unnormalized log probabilities." Aren't we ...
0
votes
1answer
4k views

A practical reason to use Cross-entropy as a error-function in Neural networks?

Cross-entropy tends to allow errors to change weights even when nodes saturate (which means that their derivatives are asymptotically close to 0.) Link Why is the above statement true? Figures ...
2
votes
0answers
152 views

The meaning of the difference of two entropy values

I want to understanding the meaning of the difference of two information entropy values. I have the following scenario. Let $x$ be a number of hours a user spend on some video sharing websites. Thus, ...
1
vote
2answers
556 views

Why the number of neurons or convolutions chosen equal powers of two?

In the overwhelming number of works devoted to the neural networks, the authors suggest arhitechure in which each layer is a numbers of neurons is power of 2 what are the theoretical reasons(...
5
votes
1answer
4k views

Why we use information gain over accuracy as splitting criterion in decision tree?

In decision tree classifier most of the algorithms use Information gain as spiting criterion. We select the feature with maximum information gain to split on. I think that using accuracy instead of ...
3
votes
1answer
1k views

How to estimate the mutual information numerically?

Suppose I have a sample {$z_i$}$_{i\in[0,N]}$ = {($x_i,y_i$)}$_{i\in[0,N]}$ which commes from a probability distribution $p_z(z)$. How can I use it to estimate the mutual information between X and Y ? ...