1
$\begingroup$

I want to build a classifier from a dataset of vectors that include exclusively boolean values. Is there any chances that my classifier might perform better if, previously to the learning, I add features that consist in combinations (with logic operators « OR » and « AND ») of the original ones or would that be pointless ?

For example I have the following dataset :

            feature A   feature B   feature C
vector 1    True        False       True
vector 2    True        True        False
vector 3    False       True        True

Imagine I have the feeling that the fact that a data has True for both feature C and feature D would typically make it go in one particular class. Before learning the datas to build the classifier, should I add to each vector en extra feature computed by the logical operation « feature B AND feature C » :

            feature A   feature B   feature C   feature D
vector 1    True        False       True        False   
vector 2    True        True        False       False
vector 3    False       True        True        True

Or would the relevance of « feature B AND feature C » be taken into account by the classifier anyway ? Does it depend on the algorithm (svm, Knn, etc. ) ?

$\endgroup$
1
$\begingroup$

Try it and see what happens. Neural networks don't have enough representational power to learn an XOR operation without at least one hidden layer, so there are definitely some interesting features you can construct with logical operations. The AND operation is equivalent to multiplication, which corresponds to interaction terms linear models.

But yeah, it does depend on the model. For example, a decision tree can learn these kinds of features on its own (although it won't necessarily). For exame, an AND operation would correspond to two tests on the same branch.

$\endgroup$
0
$\begingroup$

I would first point out that data which requires this will probably get almost always useless results with almost all well known models. But consider this especially with a large number of features: there are (n chooses k) for k=1..n combinations of various logical combination possibilities for both pure ANDs and pure ORs so you will have to very cleverly approach this. A very large Pascal's triangle could give you a better picture of what that would approach after enough features. Since you can model any Boolean formula with a series of ANDs/ORs which are then ORed/ANDed together respectively, you could just try every combination if your data set is as small as the example.

As mentioned, decision trees are the closest model to finding Boolean combinations although their strategy is pretty much counter to optimizing just the raw logic combinations without overfitting as the thresholds used their do not equate to this problem.

$\endgroup$
0
$\begingroup$

I suggest when in doubt, create the feature, and check the feature importance, which can be done by fitting a random forest.

In my own experience, I once tried creating features the way you suggest, and the features had extremely poor feature importance. Apart from that, the model performed worse than without the extra features, so it didn't go very well with me. But in the very end, you'll always have to check what happens with your data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.