-1
$\begingroup$

Suppose that we have a dataset of 2 samples :

 [{1,2,0}, {2,0,0}, {3,1,1}, {4,0,1}, {5,1,1}] 

(the last element of each row is the class variable)

If we want to reduce the variety of the column "1" we can put all values that are less or equal to "2" to a new value "1" and values that are greater than "2" to the value 2, so we would have a new dataset like :

[{1,2,0}, {1,0,0}, {2,1,1}, {2,0,1}, {2,1,1}] 

In this example we can easly deduct that there is a information gain after this transfromation, but what if we have million of sample, is there any way to measure the information gain / loss after such a transformation.

$\endgroup$
1
$\begingroup$

It will help to define what you mean by information. The usual definition is 'Shannon information', which is equivalent to the amount of uncertainty an observation resolves. If you mean something else, nothing I am about to write will apply.

Perhaps what you are looking for is the idea of mutual information. It is a way to measure the shared information between two variables - literally how much your uncertainty about one variable is reduced by knowing the value of another variable. Closely related conditional entropy, which measures the amount of uncertainty we still have about a variable given that we know the value of a second variable. Those references might be worth reading to get a sense of how information between two variables is defined in a technical sense.

In your case, your operation actually does not increase the information in your dataset. Your logical operation does not bring in any outside observation to the system, but is merely a rule-baesd transformation. It is therefore subject to the data processing inequality.

Edit: Re-reading your question, I think you are asking about the difference between I(X;Y) and I(X';Y), where I is the mutual information function, X is your input data, and Y is your target label. So basically the answer to your question is Info gain = I(X';Y) - I(X;Y). You can look up mutual information approximation methods - there are packages for most languages.

$\endgroup$
1
  • $\begingroup$ Your last edit answers perfectly to my question, my issue was to measure if I am having an info gain or loss after the transformation of data $\endgroup$ Nov 17 '17 at 23:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.