# Weighted Linear Combination of Classifiers

I am trying to build an ensemble of classifiers whereby I want my algorithm to learn a set of weights such that it can weight the outputs of different classifiers for a set of data points.

I am wondering, how would I go about learning these weights? I tried using automatic differentiation but the weights are not moving at all (no gradient information).

Does anyone know how I can fix this?

• are you asking how to do this in general? if you are just wondering about the gradients you can try a gradient-free optimizer like scipy's differential evolution which is a quite good implementation – oW_ May 14 '19 at 19:02

I don't know how to fix your automatic differentiation, but I can show you what I did (and I have seen others do too) when I wanted to achieve the same thing. You can fit a linear meta-classifier on the outputs of your classifiers that you want to ensemble. Here is the implementation from my scikit toolbox:

'''
-------------------------------------------------------
Stack Classifier - extrakit-learn

-------------------------------------------------------
'''

from sklearn.base import BaseEstimator, ClassifierMixin
from sklearn.utils.validation import check_X_y, check_array, check_is_fitted
import numpy as np

class StackClassifier(BaseEstimator, ClassifierMixin):
''' Stack Classifier

Ensemble classifier that uses one meta classifiers and several sub-classifiers.
The sub-classifiers give their output to to the meta classifier which will use
them as input features.

Parameters
----------
clfs : Classifiers who's output will assist the meta_clf, list classifier

meta_clf : Ensemble classifier that makes the final output, classifier

drop_first : Drop first class probability to avoid multi-collinearity, bool

keep_features : If original input features should be used by meta_clf, bool

refit : If sub-classifiers should be refit, bool
'''

def __init__(self, clfs, meta_clf, drop_first=True, keep_features=False, refit=True):
self.clfs = clfs
self.meta_clf = meta_clf
self.drop_first = drop_first
self.keep_features = keep_features
self.refit = refit

def fit(self, X, y):
''' Fitting of the classifier

Parameters
----------
X : array-like, shape (n_samples, n_features)
The training input samples.

y : array-like, shape (n_samples,)
The target values. An array of int.

Returns
-------
self : object
Returns self.
'''

X, y = check_X_y(X, y, accept_sparse=True)

# Refit of classifier ensemble
if self.refit:
for clf in self.clfs:
clf.fit(X, y)

# Build new tier-2 features
X_meta = build_meta_X(self.clfs, X, self.keep_features)

# Fit meta classifer, Stack the ensemble
self.meta_clf.fit(X_meta, y)

# set attributes
self.n_features_ = X.shape[1]
self.n_meta_features_ = X_meta.shape[1]
self.n_clfs_ = len(self.clfs)

return self

def predict_proba(self, X):
''' Probability prediction

Parameters
----------
X : {array-like, sparse matrix}, shape (n_samples, n_features)
The prediction input samples.

Returns
-------
y : ndarray, shape (n_samples,)
Returns an array of probabilities, floats.
'''

X = check_array(X, accept_sparse=True)
check_is_fitted(self, 'n_features_')

# Build new tier-2 features
X_meta = build_meta_X(self.clfs, X, self.keep_features)

return self.meta_clf.predict_proba(X_meta)

def predict(self, X):
''' Classification

Parameters
----------
X : {array-like, sparse matrix}, shape (n_samples, n_features)
The prediction input samples.

Returns
-------
y : ndarray, shape (n_samples,)
Returns an array of classifications, bools.
'''

X = check_array(X, accept_sparse=True)
check_is_fitted(self, 'n_features_')

# Build new tier-2 features
X_meta = build_meta_X(self.clfs, X, self.keep_features)

return self.meta_clf.predict(X_meta)

def build_meta_X(clfs, X=None, drop_first=True, keep_features=False):
''' Build features that includes outputs of the sub-classifiers

Parameters
----------
clfs : Classifiers that who's output will assist the meta_clf, list classifier

X : {array-like, sparse matrix}, shape (n_samples, n_features)
The prediction input samples.

drop_first : Drop first proba to avoid multi-collinearity, bool

keep_features : If original input features should be used by meta_clf, bool

Returns
-------
X_meta : {array-like, sparse matrix}, shape (n_samples, n_features + n_clfs*classes)
The prediction input samples for the meta clf.
'''

if keep_features:
X_meta = X
else:
X_meta = None

for clf in clfs:

if X_meta is None:
if drop_first:
X_meta = clf.predict_proba(X)
else:
X_meta = clf.predict_proba(X)[:, 1:]
else:
if drop_first:
y_ = clf.predict_proba(X)
else:
y_ = clf.predict_proba(X)[:, 1:]
X_meta = np.hstack([X_meta, y_])

return X_meta


This would allow you to use any meta-classifier, but with linear models like ridge/lasso/logistic regression it will acts as learned linear weights of your ensemble classifiers. Like this:

from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from xklearn.models import StackClassifier

X, y = make_classification(n_classes=2, n_features=4, n_samples=1000)

meta_clf = LogisticRegression(solver='lbfgs')
ensemble = [DecisionTreeClassifier(max_depth=1),
DecisionTreeClassifier(max_depth=5),
DecisionTreeClassifier(max_depth=10)]

stack_clf = StackClassifier(clfs=ensemble, meta_clf=meta_clf)
stack_clf.fit(X, y)

print('Weights:', stack_clf.meta_clf.coef_[0],' Bias: ', stack_clf.meta_clf.intercept_)


output:

Weights: [0.50017775 2.2626092  6.30510687]  Bias:  [-4.82988374]