3
$\begingroup$

I am trying to build an ensemble of classifiers whereby I want my algorithm to learn a set of weights such that it can weight the outputs of different classifiers for a set of data points.

I am wondering, how would I go about learning these weights? I tried using automatic differentiation but the weights are not moving at all (no gradient information).

Does anyone know how I can fix this?

$\endgroup$
1
  • $\begingroup$ are you asking how to do this in general? if you are just wondering about the gradients you can try a gradient-free optimizer like scipy's differential evolution which is a quite good implementation $\endgroup$ – oW_ May 14 '19 at 19:02
2
$\begingroup$

I don't know how to fix your automatic differentiation, but I can show you what I did (and I have seen others do too) when I wanted to achieve the same thing. You can fit a linear meta-classifier on the outputs of your classifiers that you want to ensemble. Here is the implementation from my scikit toolbox:

'''
-------------------------------------------------------
    Stack Classifier - extrakit-learn

    Author: Simon Larsson <larssonsimon0@gmail.com>

    License: MIT
-------------------------------------------------------
'''

from sklearn.base import BaseEstimator, ClassifierMixin
from sklearn.utils.validation import check_X_y, check_array, check_is_fitted
import numpy as np

class StackClassifier(BaseEstimator, ClassifierMixin):
    ''' Stack Classifier

    Ensemble classifier that uses one meta classifiers and several sub-classifiers.
    The sub-classifiers give their output to to the meta classifier which will use
    them as input features.

    Parameters
    ----------
    clfs : Classifiers who's output will assist the meta_clf, list classifier

    meta_clf : Ensemble classifier that makes the final output, classifier

    drop_first : Drop first class probability to avoid multi-collinearity, bool

    keep_features : If original input features should be used by meta_clf, bool

    refit : If sub-classifiers should be refit, bool
    '''

    def __init__(self, clfs, meta_clf, drop_first=True, keep_features=False, refit=True):
        self.clfs = clfs
        self.meta_clf = meta_clf
        self.drop_first = drop_first
        self.keep_features = keep_features
        self.refit = refit

    def fit(self, X, y):
        ''' Fitting of the classifier

        Parameters
        ----------
        X : array-like, shape (n_samples, n_features)
            The training input samples.

        y : array-like, shape (n_samples,)
            The target values. An array of int.

        Returns
        -------
        self : object
            Returns self.
        '''

        X, y = check_X_y(X, y, accept_sparse=True)

        # Refit of classifier ensemble
        if self.refit:
            for clf in self.clfs:
                clf.fit(X, y)

        # Build new tier-2 features
        X_meta = build_meta_X(self.clfs, X, self.keep_features)

        # Fit meta classifer, Stack the ensemble
        self.meta_clf.fit(X_meta, y)

        # set attributes
        self.n_features_ = X.shape[1]
        self.n_meta_features_ = X_meta.shape[1]
        self.n_clfs_ = len(self.clfs)

        return self

    def predict_proba(self, X):
        ''' Probability prediction

        Parameters
        ----------
        X : {array-like, sparse matrix}, shape (n_samples, n_features)
            The prediction input samples.

        Returns
        -------
        y : ndarray, shape (n_samples,)
            Returns an array of probabilities, floats.
        '''

        X = check_array(X, accept_sparse=True)
        check_is_fitted(self, 'n_features_')

        # Build new tier-2 features
        X_meta = build_meta_X(self.clfs, X, self.keep_features)

        return self.meta_clf.predict_proba(X_meta)

    def predict(self, X):
        ''' Classification

        Parameters
        ----------
        X : {array-like, sparse matrix}, shape (n_samples, n_features)
            The prediction input samples.

        Returns
        -------
        y : ndarray, shape (n_samples,)
            Returns an array of classifications, bools.
        '''

        X = check_array(X, accept_sparse=True)
        check_is_fitted(self, 'n_features_')

        # Build new tier-2 features
        X_meta = build_meta_X(self.clfs, X, self.keep_features)

        return self.meta_clf.predict(X_meta)

def build_meta_X(clfs, X=None, drop_first=True, keep_features=False):
    ''' Build features that includes outputs of the sub-classifiers

    Parameters
    ----------
    clfs : Classifiers that who's output will assist the meta_clf, list classifier

    X : {array-like, sparse matrix}, shape (n_samples, n_features)
        The prediction input samples.

    drop_first : Drop first proba to avoid multi-collinearity, bool

    keep_features : If original input features should be used by meta_clf, bool

    Returns
    -------
    X_meta : {array-like, sparse matrix}, shape (n_samples, n_features + n_clfs*classes)
                 The prediction input samples for the meta clf.
    '''

    if keep_features:
        X_meta = X
    else:
        X_meta = None

    for clf in clfs:

        if X_meta is None:
            if drop_first:
                X_meta = clf.predict_proba(X)
            else:
                X_meta = clf.predict_proba(X)[:, 1:]
        else:
            if drop_first:
                y_ = clf.predict_proba(X)
            else:
                y_ = clf.predict_proba(X)[:, 1:]
            X_meta = np.hstack([X_meta, y_])

    return X_meta

This would allow you to use any meta-classifier, but with linear models like ridge/lasso/logistic regression it will acts as learned linear weights of your ensemble classifiers. Like this:

from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from xklearn.models import StackClassifier

X, y = make_classification(n_classes=2, n_features=4, n_samples=1000)

meta_clf = LogisticRegression(solver='lbfgs')
ensemble = [DecisionTreeClassifier(max_depth=1), 
            DecisionTreeClassifier(max_depth=5), 
            DecisionTreeClassifier(max_depth=10)]

stack_clf = StackClassifier(clfs=ensemble, meta_clf=meta_clf)
stack_clf.fit(X, y)

print('Weights:', stack_clf.meta_clf.coef_[0],' Bias: ', stack_clf.meta_clf.intercept_)

output:

Weights: [0.50017775 2.2626092  6.30510687]  Bias:  [-4.82988374]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.