Differences in OvO SVC decision_function and predict methods in Scikit learn package

Question

I have observed that there are differences in the implementation of the decision_function and the predict methods the one versus one multi class implementation of SVC in the sci-kit learn package. Is there a way to align this issue or is this indeed a bug in the implementation?

I was solving a multi classification problem with 11 classes with around 36 initial feature variables (160 features after some transformation e.g. scaling, missing value indicators etc.) and over 6,000 observations using the OvO linear SVC implemented in sci-kit learn. In cases where there are ties in votes for 2 or more of the 55 classifiers generated by the OvO SVC, the decision_function method uses confidences to break the ties in this case (see the source code from line 479 -521 here ). However, the predict method implements LIBSVM and a close look at the C++ code reveals that in the case of ties, the first runtime index in the loop is kept as the decision for the OvO linear SVC (see the source code from line 2865 to 2899 here).

For example, if there is a tie between class 5 and 7 with 17 votes each, class 5 will be taken to be the decision of the predict function since 5 is taken as the index before 7 in the for loop. I am not allowed to share the actual examples i have, but it is clear from the implementation that these cases my arise simply because this confidences are not implemented in LIBSVM.

I expected the decisions/predictions of both the decision_function method and the predict method to give the same results for the same parameters of the OvO SVC. In this particular multi classification case, whenever there are ties, there are some cases where the decision_function method give different results to those of the predict method. In my, actually the implementation of the decision function is more correct since it involves some quantified justification for making the decision in case of ties.

Is this really how the implementation should be or is this a bug in the implementation? Would it be possible to implement this confidences in LIBSVM package as part of the standard implementation ? How would these custom modifications look like ? I mean somehow i don't think i am the first person to see this issue so i am quite sure there is something i am missing, but i have not found this case discussed anywhere in this forum or any other mains stream python forum. I would appreciate any help and clarification from the experts here.

Answer 1

It is expected that the decision_function and predict methods of the OneVsOneClassifier class in scikit-learn may give different results in cases of ties. This is because the decision_function method returns the distances of the samples to the decision boundary for each class, while the predict method returns the class labels themselves.

The decision_function method uses the distances to the decision boundary to break ties, while the predict method uses the order of the classes as determined by the classes_ attribute of the classifier. This attribute is determined when the classifier is fit, and it is the order in which the class labels are encountered during training. Therefore, the order of the classes in the classes_ attribute determines the class that is chosen in cases of ties by the `predict method.

It is not possible to modify the implementation of the LIBSVM library, which is used by scikit-learn, to use the confidences in the decision function to break ties. However, you can use the decision_function method to make predictions, which will break ties using the confidences.

Alternatively, you can use a different classifier that does not have this issue, such as a classifier that uses probability estimates to make predictions, such as the SVC class with the probability parameter set to True. In this case, the classifier will return the probabilities for each class, and you can use these probabilities to break ties and make predictions.

I hope this helps clarify the difference between the decision_function and predict methods and how they handle cases of ties. Let me know if you have any further questions.