0
$\begingroup$

I have 3 sklearn models which I use to predict a probability score for a binary classification problem. I want to create a weighted average score of all the predictions made by these models. I am stuck at how to find the optimum weights.

I have tried to create a weighted average method that would help me:

def weighted_average(prob: dict, weights: dict = base_weights):
    '''Weighted average of all probabilities
    Prob Dict structure: {
    'mfcc': probability of spoof from MFCC Model,
    'lfcc': probability of spoof from LFCC Model,
    'gfcc': probability of spoof from GFCC Model
    }
    
    returns weighted average of probability
    '''
    num = prob['mfcc']*weights['mfcc'] + prob['lfcc']*weights['lfcc'] + prob['gfcc']*weights['gfcc']
    denom = weights['mfcc'] + weights['lfcc'] + weights['gfcc']
    return num / denom

In order to find the optimum weights (I'm optimizing accuracy_score), I have tried the following:

  1. Exhaustive search over a range.
  2. Tried fitting a Logistic regression model with X as the accuracy scores and Y as 0|1.

In the end, the goal is to get an accuracy score and not 0 or 1.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Sklearn has built-in ensembling classes for example voting classifier for this job.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Actually, the 3 models are trained on 3 different features, so VotingClassifer may not be a valid use case here. $\endgroup$
    – ShubhankarKG
    May 4, 2022 at 5:13
  • $\begingroup$ Then it will be a bit more work. But why doesn't your approach #2 work? $\endgroup$
    – lpounng
    May 4, 2022 at 6:12
  • $\begingroup$ Actually, when we train the model, we have accuracies from 3 features as X, and we train a classifier. But in the use case of this model, we want to get a combiner probability score, not a 0 or 1, which does not work $\endgroup$
    – ShubhankarKG
    May 5, 2022 at 2:37
  • $\begingroup$ How about collecting the outputs of the 3 models, as input and train a final model which can produce probability score? You do not have to over-complicated yourself on how to combine. $\endgroup$
    – lpounng
    May 5, 2022 at 3:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.