0
$\begingroup$

I have few pre-trained blackbox models that are used for classification tasks. I want to know what is the best way to combine these models into a single classifier that does not require any re-training of these models.

Edit: The models take input $X$, and predicts a single class $\hat{Y}$ out of several possible classes. Some models are able to predict correct class in some instances of $X_i$ while other predict wrong class and vice-versa. My goal is to use these outputs $\hat{Y}_i$ as input features to new meta-modal which is trained on these array of predicted classes as input along with ground-truth $Y$.

$$M(\hat{Y}_i) \rightarrow Y$$

Two options were majority voting classifier and stacking. However this methods require to retrain underlying classifiers. My classifiers are blackbox so it there a better way rather then simple max voting?

$\endgroup$
5
  • $\begingroup$ How do you know the models have different skills? Any other details you can share? Like is one an SVM blackbox and another a neural net blackbox? That can make a difference for how an ensemble exploits their respective strengths. As stated, the OP suggests that your current task is to train a model that accepts N + 6 features, viewing each model output as just-another-feature. // Tell us about the inputs and the outputs. Does it just emit a hard 0 / 1 classification labels, or do you get a soft 0 .. 1 probability? Are the inputs categorical or continuous? $\endgroup$
    – J_H
    Jan 14 at 18:38
  • $\begingroup$ @J_H the are transformer based blackboxes. The input to the model is an array of features X and the models perform mult-class classification y^. In some instances some models predict correct class while others predict wrong class and vice versa so I want to combine them by training a meta-model so that it captures such short-comings of the model based on the outputs as its input features y^s and picks correct output from it. The model emits hard classification. I get the concrete label in the end $\endgroup$ Jan 15 at 6:53
  • $\begingroup$ @J_H I have updated the question. $\endgroup$ Jan 15 at 7:29
  • 1
    $\begingroup$ That’s too bad, that we get a hard classification output. If the transformers output a probability, a confidence in the results, that would make it easier for the meta-model to steer its response toward results having greater confidence. // Notice that a transformer which outputs a single classification drawn from K different classes, could also output K different boolean values with associated confidence. That is, the output would be arg max of the K different probabilities. This would be more suitable as input to a meta-model. $\endgroup$
    – J_H
    Jan 15 at 13:24
  • $\begingroup$ @J_H This makes sense. I took different approach by encoding the classes to numerical values. I believe you method of utilizing confidence could have been much better, but the model doesnt output confidence levels. $\endgroup$ Jan 16 at 13:36

1 Answer 1

2
$\begingroup$

After some more reviewing I came across the following solution:

enter image description here

So basically Logistic regression's input is supposed to be numerical values, while the Neural networks (NNs) that I have gave string-based classes. I simply encoded each class to numeric values, (I had 10 classes so 1,2,3..10) and fed these as input to "multi-class logistic regression" and I was able to furthur refine the output from several NNs closer to the ground-truth.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.