I have 3 sklearn models which I use to predict a probability score for a binary classification problem. I want to create a weighted average score of all the predictions made by these models. I am stuck at how to find the optimum weights.
I have tried to create a weighted average method that would help me:
def weighted_average(prob: dict, weights: dict = base_weights):
'''Weighted average of all probabilities
Prob Dict structure: {
'mfcc': probability of spoof from MFCC Model,
'lfcc': probability of spoof from LFCC Model,
'gfcc': probability of spoof from GFCC Model
}
returns weighted average of probability
'''
num = prob['mfcc']*weights['mfcc'] + prob['lfcc']*weights['lfcc'] + prob['gfcc']*weights['gfcc']
denom = weights['mfcc'] + weights['lfcc'] + weights['gfcc']
return num / denom
In order to find the optimum weights (I'm optimizing accuracy_score), I have tried the following:
- Exhaustive search over a range.
- Tried fitting a Logistic regression model with X as the accuracy scores and Y as 0|1.
In the end, the goal is to get an accuracy score and not 0 or 1.