3
$\begingroup$

Probabilistic classifiers look really good because they give you more information than deterministic ones i.e. estimated probabilities of class memberships rather than just which class the model thinks an individual datum should belong to.

So in what circumstances would you choose a deterministic classifier rather than a probabilistic one?

Improve this question
$\endgroup$
2

1 Answer 1

Reset to default
3
+50
$\begingroup$

It is worth considering using a "deterministic" classifier if all of the following conditions are met:

  1. the false-positive and false-negative misclassification costs are known beforehand and are either fixed, or you don't mind retraining your model when the change;
  2. the relative class frequencies in operation are known beforehand and are either fixed, or you don't mind retraining your model when they change;
  3. ou don't need a "reject" option (although there are ways around that).

In those situations, you might want to use a classifier like the Support Vector Machine that focusses on solving the classification problem directly. The reason for this is that a probabilistic classifier tries to predict the probability accurately everywhere, and will expend modelling resources doing so. A discrete/deterministic classifier, on the other hand, only focuses resources on estimating the position of one particular contour of probability, as that gives the optimal decision boundary, so in principle it can make better use of the available data.

The nice thing about probabilistic classifiers is that you can adjust for changes in misclassification costs, or changes in relative class frequencies, or implement a "reject" option easily and without having to retrain the model. The downside is they make slightly less good use of the data as they consider features of the data distribution that are not relevant to the optimal classification, see my example of that here: https://stats.stackexchange.com/questions/312780/why-is-accuracy-not-the-best-measure-for-assessing-classification-models/538524#538524 . So if you don't need/want any of those nice properties of probabilistic classifiers, you might get better results using a discrete classifier (and the success of the SVM gives evidence that is true in a variety of practical applications).

In short, have both sets of tools in your data science toolbox as both are useful.

Improve this answer
$\endgroup$
13
  • 1
    $\begingroup$ If the difference in probability is too small, for instance P(A|X) = 0.45; P(B|X) = 0.46 and P(C|X) = 0.09, then that means the classifier is pretty sure it isn't class C, but it is not very confident between whether it is class A or class B. Say those are diagnoses of similar cancers with different treatment regimes. In that case it might be better to choose not to classify at this stage and gather more information to make a more confident, better choice later, to give a better chance of picking the right treatment. It is difficult to do that with a deterministic classifier. $\endgroup$
    – Dikran Marsupial
    Aug 18, 2021 at 7:15
  • 1
    $\begingroup$ In Baye's rule, the probability of class membership $(P(C_i|X) \propto P(X|C)P(C_i)$, so it depends on the class frequency. This means the classifier will learn to assign probabilties of class membership for the class frequencies in the training set, and if the operational class frequencies are different, the probabilities will be wrong. If they are fixed, you can accommodate that by resampling the training data or weighting the training patterns. $\endgroup$
    – Dikran Marsupial
    Aug 18, 2021 at 9:49
  • 1
    $\begingroup$ In e.g. a medical diagnosis test, the cost of classifying the patient as having cancer when they don't is fairly small, they will be worried, but the error will be spotted when further tests are carried out. The costs of a false negative - telling them they don't have cancer when they do is far higher as they may die or become very severely ill before the error is detected. So we should build these costs into the training criterion (or in setting the decision boundary for probabilistic classifiers) $\endgroup$
    – Dikran Marsupial
    Aug 19, 2021 at 9:35
  • 1
    $\begingroup$ @hirschme $P(X|C)$ describes how the data for patterns belonging to a particular class are distributed. In parametric classifiers (e.g. Naive Bayes or Gaussian classifiers), $P(X|C)$ appears explicitly in the model, but most modern classifiers are "non-parametric" which means that the likelihood is not explicitly written down in the model formulation. If we are just interested in classification, we don't need to model parts of $P(X|C)$ that don't tell us anything about the decision boundary, so non-parametric models can be simpler. $\endgroup$
    – Dikran Marsupial
    May 11, 2022 at 9:34
  • 1
    $\begingroup$ @hirschme yes, but even if the likelihood is used in the training criterion, it may not be explicitly visible in the structure of the model. The likelihood is usually frames as $P(X,Y|Model)$, as the density of $X$ is only included via the sampling of the training patterns. Note that only $Y$ appears explicitly in the cross-entropy, X only appears indirectly via the model output. $\endgroup$
    – Dikran Marsupial
    May 12, 2022 at 1:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.