I've developed an algorithm to define conditional entropy for feature selection in text classification. I'm following the formula at Machine Learning from Text by Charu C. Aggarwal (5.2.2). The author mentions that Conditional Entropy values are between (0, log(number of classes)) in which my case is (0, 0.6931472).

The author also mentions that features with the largest values can be removed, but he don't give further information about the criteria to define 'largest' (is it only the max value of entropy or a set of the largest entropy values?)

Have you ever guys applied Conditional Entropy for feature selection? If so, based on results, what criteria was used to define features to be removed.

Here a summary of my Conditional Entropy results:

 Min.   :0.5701  
 1st Qu.:0.6562  
 Median :0.6563  
 Mean   :0.6558  
 3rd Qu.:0.6564  
 Max.   :0.6564 

Individual feature selection methods assign a numerical value to every feature so that features can be ranked according to this value. The calculated value is chosen to represent how much the feature contributes to knowing the label/response variable: common choices are conditional entropy, but also information gain or correlation.

The actual values assigned to the features are not really useful on their own, what matters is the ordering of the features according to this value. Thus the standard method for selecting the features is not to choose a particular threshold on the value, but simply to choose the number of features one wants to obtain.

Example: in a text classification task, there are 1000 documents and a vocabulary of 20000 unique words as candidate features. Using all the words would certainly cause overfitting, so we decide to use only 100 words as features. We can calculate the conditional entropy of every word with respect to the label, and then select the bottom 100 words according to the corresponding ranking as features (the 19900 other words are ignored).

Since individual feature selection is very efficient, it's often possible (and a good idea) to try a range of values as the number of features, and train/test the model for each of these values. This way one can experimentally determine the optimal number of features (the one which maximizes performance on the data). Note that this a form of hyper-parameter tuning, and therefore one has to use a validation set for the tuning stage, and then the final model (with the selected optimal number of features) should be applied on a fresh test set.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.