There is a general recommendation that algorithms in ensemble learning combinations should be different in nature. Is there a classification table, a scale or some rules that allow to evaluate how far away are the algorithms from each other? What are the best combinations?
In general in an ensemble you try to combine the opinions of multiple classifiers. The idea is like asking a bunch of experts on the same thing. You get multiple opinions and you later have to combine their answers (e.g. by a voting scheme). For this trick to work you want the classifiers to be different from each other, that is you don't want to ask the same "expert" twice for the same thing.
In practice, the classifiers do not have to be different in the sense of a different algorithm. What you can do is train the same algorithm with different subset of the data or a different subset of features (or both). If you use different training sets you end up with different models and different "independent" classifiers.
There is no golden rule on what works best in general. You have to try to see if there is an improvement for your specific problem.
As a rule of thumb I always propose three different options:
- Use a bagging learning technique, similar to that one followed by Random Forest. This technique allows the training of 'small' classifiers which see a small portion of the whole data. Afterwards, a simple voting scheme (as in Random Forest) will lead you to a very interesting and robust classification.
- Use any technique related to fusioning information or probabilistic fusion. This is a very suitable solution in order to combine different likelihoods from different classifiers.
- My last suggestion is the use of fuzzy logic, a very adequate tool in order to combine information properly from a probabilistic (belonging) perspective.
The selection of specific methods or strategies will depend enormously on the data.