1
$\begingroup$

I need to train several Logistic regression models on a different set of data (with a different set of labels):

Train data of model 1:
[X1, X2, X3] -> y1
[X4, X5, X6] -> y2
...
Train data of model N:
[X99, X101, X102] -> y8
[X103, X104] -> y9

As a model, I choose Logistic regression with a One-vs-Rest train scheme. When I get an input from the user, I need to determine which label was most probable (from all N models). So what I am trying to do is query trained models with validation data and calculate mean MU and standard deviation STD. Based on this value I will normalize predictions to be comparable between each other - is the assumption correct?

# normalization
normalized_predictions = [(pr - g["pos_mu"]) / g["pos_std"] for g, pr in zip(self.gaussians.values(), prediction)]

# example output of each Logistic regression model
output.append({
                    'answer': self.idx2label[idx],
                    'confidence': normalized_predictions[idx],
                    'hit': ""
})

...
# query and normalize - normalization is shown above
responses = query_log_reg_model_and_normalize() 

# sort based on confidence value
responses = sorted(responses, key=lambda r: r['confidence'], reverse=True)

# return most probable
return responses
``` 
$\endgroup$
6
  • $\begingroup$ I don't think it's possible to answer whether it's correct or not to normalize the predictions. It might make sense, but imho the only way to know is to evaluate the two options (with/without normalization) on a validation test. $\endgroup$
    – Erwan
    Mar 26 at 11:52
  • $\begingroup$ But if I don't do any kind of normalization then there is inconsistency - imagine a scenario when first model is trained on 2 classes and the other on 3. I will input an example from the second model for the first class, then, for example, the first model return ([0.51, 0.49]) but the other ([0.5,0.25,0.25]). If I will join the prediction without normalization then it will lead to first model but the second one is more confident (0.5 is better compared to 0.25 than 0.59 to 0.49) $\endgroup$
    – United121
    Mar 26 at 12:12
  • $\begingroup$ Oh right sorry I didn't think this through. Mmh then it would be harder than normalizing, because a model can be completely wrong from the start if it doesn't have the correct answer at all, right? $\endgroup$
    – Erwan
    Mar 26 at 21:12
  • $\begingroup$ I thought a bit about this and I don't think this approach can work on its own, because what you obtain is a conditional probability: it's $p(y=X1| y \in \{ X1,X2,X3\})$. The problem is that this way there's no general way to calculate any probability over the full set of classes, especially if the classes in 2 different models are always different. Even with normalization the predicted probabilities are not comparable: for example if X1 is much more likely than X2 or X3 while X6 is only slightly more likely that X4 or X5, you're likely to have p(X1)>p(X6) but X6 could be the right answer. $\endgroup$
    – Erwan
    Mar 26 at 22:37
  • $\begingroup$ I think that the normalization (and approach above) would work if the model was not "overconfident" with his prediction (but this condition is usually not satisfied) -then it would not predict X1 with high probability if the true class is X6. It would be nice if the model was following the "open-world assumption" and let the model say that he is not sure. $\endgroup$
    – United121
    Mar 28 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.