2
$\begingroup$

while comparing two different algorithms to feature selection I stumbled upon the follwing question:

For a given dataset with a discrete class variable we want to train a naive bayes classifier. We decide to conduct feature selection during preprocessing using naive bayes in a wrapper approach. Does this method of feature selection consider the size of the used feature subsets?

Naive Bayes Classification

When considering how NB classifies a given instance, the size of the feature subset being used for classification only influences the number of parts that the product of the conditional dependencies has but that does not make a difference, or does it?

It'd be great if someone could offer a solid explanation since for me it's more of a gut feeling at the moment.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Short Version

The difference is in the trade-off between complexity of model and how hard it will be to train. The more states your variable can take, the much more data you'll need to train it. It sounds like you're reducing the feature space through some front-end processing.

Example

Context:

Say you're using NBC to model if you go outside given the high temperature that day. Let's say temperature is given to you as an int in Celsius which just happened to range between 0 and 40 degrees. Over one year of running the experiment - recording the temperature and if you went outside - you'll have 365 data points.

Internal Representation:

The internal structure the NBC uses will be a matrix with 82 values (41 values for your one input variable * 2 because you have two states in your output class). That means each bin will have an average of 365/82 = 8.9ish samples. In practice, you'll probably see a few bins with lots more samples than this and a many bins with 0 samples.

A Pitfall

Say you saw 8 cases at a temperature of 5 C (all of which you stayed inside) and nothing at a temp of 3 or 4 C. If, after the model is built, you 'ask it' what class a temp of 4 C should be in. It won't know. Intuitively, we'd say "stay inside", but the model won't. One way to fix this is to bin the classes 0,1,2,... into larger groups of temperatures (i.e., class 0 for temp 0-3, class 1 for temp 4-7, etc). The extreme case of this would be two temperature states "high" and "low". The actual cut-off should depend more on the data observed, but one such scheme is converting temperatures within 0-20 to "low" and 21-40 is "high".

Discussion

This front end processing seems to be what you're talking about as the "wrapper" around the NBC. The result of our "high" and "low" binning scheme will result in a much smaller model (2 input states and 2 output classes gives you a 2*2 = 4 number of classes. This will be way easier to process and will take way less data to confidently train at the expense of complexity. One example of a drawback of such a scheme is: Say the actual pattern is that you love spring and fall weather and only go outside when it's not too hot or not too cold. So your outside adventures are evenly split across the upper part of the "low" class and the lower part of the "high" class giving you a really useless model. If this were the case, a 3-class bin scheme of "high", "medium", and "low" would be best.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I think if you model temperature, than a numeric variable should be considered. In you specific case, I would think that building an empirical distribution from conditioned data would work much better than considering temperature as a nominal variable. $\endgroup$
    – rapaio
    Commented Jan 7, 2015 at 7:56
  • $\begingroup$ @rapaio can you expand a bit? The only definitions of 'empirical distribution' I know are either doing the same thing I did in the example or are not applicable to the current problem. $\endgroup$
    – TheGrimmScientist
    Commented Jan 7, 2015 at 19:40
  • $\begingroup$ You built an empirical distribution for a nominal variable. My proposal was to build an E.D. for a continuous variable, for example a KDE ( see en.wikipedia.org/wiki/Kernel_density_estimation ). Using a KDE the probability from each sample value 'concentrated' in each point is 'spreaded' in its neighburhood. The result is that for each possible temperature value you will get a probability from the sample points in its neghbourhood and you don't need to build intervals, which are arbitrary and thus have good chances to loose a lot of valuable information. $\endgroup$
    – rapaio
    Commented Jan 8, 2015 at 6:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.