What should I care about while stacking as an ensemble method?

Question

I'm using SMO, Logistic Regression, Bayesian Network and Simple CART algorithms for classification. Results form WEKA:

Algorithm               Sensitivity (%)       Specificity (%)         Overall accuracy (%)
Bayesian Network            57.49                 76.09                    65.24
Logistic Regression         64.73                 69.86                    66.87
SMO                         54.32                 79.20                    64.69
Simple CART                 71.88                 61.51                    67.56

SMO gives the best result for my classification problem, since it correctly classify the 79.20% of the class which is important for me. I want to increase this accuracy by stacking. I tried to combine some of them. In most of the cases I couldn't increase the accuracy but stacking SMO with Logistic Regression made a little increment in accuracy.

How can I explain why stacking SMO with Logistic Regression is better than others?

Is there any generalization such as combining tree classifiers gives good result in stacking? What should I care about while stacking?

EDIT:

                                Bayesian Network    Logistic Reg.   SMO         CART 
Kappa statistic                   0.3196             0.3367         0.3158      0.3335 
Mean absolute error               0.3517             0.4164         0.3531      0.4107 
Root mean squared error           0.5488             0.4548         0.5942      0.4547 
Relative absolute error (%)      72.3389              85.65        72.6299      84.477 
Root relative squared error (%) 111.3076            92.2452       120.5239     92.2318 
Weighted Avg. of F-Measure        0.653               0.671          0.676     92.2318 
ROC Area                          0.725               0.727          0.668       0.721

Total number of instance is 25106. 14641 of them is class a, and 10465 of them belong to class b.

=== Confusion Matrix of Simple CART ===
     a     b   <-- classified as
 10524  4117 |     a = 0
  4028  6437 |     b = 1

=== Confusion Matrix of SMO ===

    a    b   <-- classified as
 7953 6688 |    a = 0
 2177 8288 |    b = 1

=== Confusion Matrix of Logistic Regression ===

    a    b   <-- classified as
 9477 5164 |    a = 0
 3154 7311 |    b = 1

Since SMO is successful at class b and CART is successful at class a, I tried to ensemble these two algorithms. But I couldn't increase the accuracy. Then I tried to combine SMO with Logistic Regression, the accuracy is increased a little bit. Why ensembling SMO with Logistic Regression is better than ensebling SMO with CART, is there any explanation?

Answer 1

To directly answer your question about stacking: you should care about minimizing 1) bias, and 2) variance. This is obvious, but in practice this often comes down to simply having models which are "diverse". (I apologize that link is behind a paywall, but there are a few others like it and you may well find it other ways)

You don't want ensembles of like-minded models - they will make the same mistakes and reinforce each other.

In the case of stacking, what is happening? You are letting the outputs of the probabilistic classifiers on the actual feature input become the new features. A diverse set of classifiers which can in any way give signals about edge cases is desirable. If classifier 1 is terrible at classes A, B, and C but fantastic at class D, or a certain edge case, it is still a good contribution to the ensemble.

This is why neural nets are so good at what they do in image recognition - deep nets are in fact recursive logistic regression stacking ensembles! Nowadays people don't always use the sigmoid activation and there are many layer architectures, but it's the same general idea.

What I would recommend is trying to maximize the diversity of your ensemble by using some of the similarity metrics on the classifiers' prediction output vectors (ie, Diettrich's Kappa statistic) in training. Here is another good reference.

Hope that helps.

Answer 2

Read the following by MLWave: http://mlwave.com/kaggle-ensembling-guide/

This is very good starting point to stacking / ensembles.