I found packages being used to calculating "Information Gain" for selecting main attributes in C4.5 Decision Tree and I tried using them to calculating "Information Gain".

But the results of calculation of each packages are different like the code below.

> IG.CORElearn <- attrEval(In_Occu ~ In_Temp+In_Humi+In_CO2+In_Illu+In_LP+Out_Temp+Out_Humi, dataUSE1, estimator = "InfGain")
> IG.RWeka     <- InfoGainAttributeEval(In_Occu ~ In_Temp+In_Humi+In_CO2+In_Illu+In_LP+Out_Temp+Out_Humi, dataUSE1)
> IG.FSelector <- information.gain(In_Occu ~ In_Temp+In_Humi+In_CO2+In_Illu+In_LP+Out_Temp+Out_Humi,dataUSE1)

> IG.CORElearn
   In_Temp    In_Humi     In_CO2    In_Illu      In_LP   Out_Temp   Out_Humi 
0.04472928 0.02705100 0.09305418 0.35064927 0.44299167 0.01832216 0.05551973 
> IG.RWeka
   In_Temp    In_Humi     In_CO2    In_Illu      In_LP   Out_Temp   Out_Humi 
0.11964771 0.04340197 0.12266724 0.38963327 0.44299167 0.03831816 0.07705798 
> IG.FSelector
In_Temp       0.08293347
In_Humi       0.02919697
In_CO2        0.08411316
In_Illu       0.27007321
In_LP         0.30705843
Out_Temp      0.02656012
Out_Humi      0.05341252

Why do the results of calculation of each packages be different? And Which one is right?


This is not a complete answer to your question, but I can explain at least part of the problem. Since you do not provide your data, I cannot reproduce your results. However, it is easy to demonstrate the same problem with other data. I will use the well-known iris data set that is provided with R and Weka and is easy to access.

The same problem is apparent with the iris data.

IG.CORElearn <- attrEval(Species ~ ., data=iris,  estimator = "InfGain")
IG.RWeka     <- InfoGainAttributeEval(Species ~ ., data=iris,)
IG.FSelector <- information.gain(Species ~ ., data=iris,)

Sepal.Length  Sepal.Width Petal.Length  Petal.Width 
   0.5572327    0.2831260    0.9182958    0.9182958 

Sepal.Length  Sepal.Width Petal.Length  Petal.Width 
   0.6982615    0.3855963    1.4180030    1.3784027 

Sepal.Length       0.4521286
Sepal.Width        0.2672750
Petal.Length       0.9402853
Petal.Width        0.9554360

As with your example, all three packages give completely different results.

One Issue: Units (Base of Logarithm)

If you look at the documentation for information.gain in FSelector, you will see this parameter description:

Unit for computing entropy (passed to entropy). Default is "log".

Following that trail, we look at the description of the entropy function and see:

the unit in which entropy is measured. The default is "nats" (natural units). For computing entropy in "bits" set unit="log2".

If we override the default and calculate IG using unit="log2" we get

IG.FSelector2 <- information.gain(Species ~ ., data=iris, unit="log2")
Sepal.Length       0.6522837
Sepal.Width        0.3855963
Petal.Length       1.3565450
Petal.Width        1.3784027

Notice that now the values for Information Gain agree with RWeka for Sepal.Width and Petal.Width. Part of the difference was simply using a different base for the logarithm. RWeka uses base 2 (entropy measured in bits). By default, FSelector uses base e, but allows you to change the base and get some of the same results. It appears from the documentation that neither RWeka nor CORElearn let you select the base for the logarithm.

But it is almost stranger that once we get RWeka and FSelector in the same units, they agree on two variables, but not the other two. There is still something else going on.


Adding to [G5W's answer][https://datascience.stackexchange.com/a/16249/29575] that FSelector (and possibly the other implementations as well) performs a discretization of features before calculating scores.

For FSelector, this is done in the file selector.info.gain.R. You can check out the discretization with FSelector:::discretize.all. This step removes information to the extent that the ordering of features is altered.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.