I have observed that there are differences in the implementation of the decision_function and the predict methods the one versus one multi class implementation of SVC in the sci-kit learn package. Is there a way to align this issue or is this indeed a bug in the implementation?
I was solving a multi classification problem with 11 classes with around 36 initial feature variables (160 features after some transformation e.g. scaling, missing value indicators etc.) and over 6,000 observations using the OvO linear SVC implemented in sci-kit learn. In cases where there are ties in votes for 2 or more of the 55 classifiers generated by the OvO SVC, the decision_function method uses confidences to break the ties in this case (see the source code from line 479 -521 here ). However, the predict method implements LIBSVM and a close look at the C++ code reveals that in the case of ties, the first runtime index in the loop is kept as the decision for the OvO linear SVC (see the source code from line 2865 to 2899 here).
For example, if there is a tie between class 5 and 7 with 17 votes each, class 5 will be taken to be the decision of the predict function since 5 is taken as the index before 7 in the for loop. I am not allowed to share the actual examples i have, but it is clear from the implementation that these cases my arise simply because this confidences are not implemented in LIBSVM.
I expected the decisions/predictions of both the decision_function method and the predict method to give the same results for the same parameters of the OvO SVC. In this particular multi classification case, whenever there are ties, there are some cases where the decision_function method give different results to those of the predict method. In my, actually the implementation of the decision function is more correct since it involves some quantified justification for making the decision in case of ties.
Is this really how the implementation should be or is this a bug in the implementation? Would it be possible to implement this confidences in LIBSVM package as part of the standard implementation ? How would these custom modifications look like ? I mean somehow i don't think i am the first person to see this issue so i am quite sure there is something i am missing, but i have not found this case discussed anywhere in this forum or any other mains stream python forum. I would appreciate any help and clarification from the experts here.