I am trying to build a binary classification system using different classification algorithms like random forests, support vector machines, AdaBoost. I want to use the output of these classifiers to visualize a score. For example, when using random forests, I would like to use the probability of a sample belonging to class A to build a score from 0 to 100. Given that random forests output a probability (from 0 to 1) using it as the score is intuitive (I would just multiply it by 100). However, given that SVMs output a classification but not a probabilistic output (i.e. distance to the hyperplane, but not probability), would it be legitimate to use the distance to the hyperplane as some sort of "pseudo probability"? I would, for example, do max-min scale on the distance to the hyperplane for all samples so all distances are scaled from 0 to 1.

I want to be sure that I can use the distance to the hyperplane as a pseudo-probability and that this pseudo-probability is comparable to the probability to belong to a given class outputted by the random forest. For example, that a sample with a probability of .80 of belonging to class A is the same than another sample of a (min-max transformed) probability of belonging to class A according to the SVM.

  • $\begingroup$ I haven't read the paper, but you might want to have a look at this (Support Vector Machines as Probabilistic Models, Franc et. al. 2011) $\endgroup$ – timleathart Dec 8 '16 at 1:06

One standard way to obtain a "probability" out of a SVM is to use Platt scaling. See, e.g., this Wikipedia page and this question on Stats.SE. Platt scaling involves fitting a logistic regression model to predict the "probability", based on the distance to the hyperplane.

| improve this answer | |
  • $\begingroup$ After applying Platt scaling is it legitimate to compare the probability output of a random forest and a SVM? Say, if you have an RF and SVM, and they both output 0.8 of probability to belong to class A, does it mean the same in both cases? Do both models have to share the same variable to mean the same? $\endgroup$ – user3824382 Dec 9 '16 at 16:01
  • 1
    $\begingroup$ @user3824382, in principle, yes, they are comparable -- a probability is a probability. In practice, I don't know whether I'd recommend it; it depends on what you're trying to achieve. $\endgroup$ – D.W. Dec 9 '16 at 16:40
  • $\begingroup$ My team's objective is to give users (for example, physicians) a tool to identify the patients with the highest risk to have a disease, and rank them. So, would it be legitimate to rank a patient with probability 0.8 to have a disease (as defined by a random forest model) higher than another with probability 0.7 (as defined by an SVM). $\endgroup$ – user3824382 Dec 9 '16 at 18:40
  • $\begingroup$ @user3824382, that sounds like something you should ask as a separate question using the 'Ask Question' button in the upper-right. (It's not clear to me why you would use different classifiers for two different patients, rather than using the same classifier for all patients.) $\endgroup$ – D.W. Dec 9 '16 at 18:51

As @D.W. mentioned, you can use Platt scaling.

However, keep in mind that the probability estimates may be inconsistent with the prediction.
From sklearn documentation:

Needless to say, the cross-validation involved in Platt scaling is an expensive operation for large datasets. In addition, the probability estimates may be inconsistent with the scores, in the sense that the "argmax" of the scores may not be the argmax of the probabilities. (E.g., in binary classification, a sample may be labeled by predict as belonging to a class that has probability <½ according to predict_proba.) Platt's method is also known to have theoretical issues. If confidence scores are required, but these do not have to be probabilities, then it is advisable to set probability=False and use decision_function instead of predict_proba.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.