0
$\begingroup$

I do not know how to interpret the result of:

prediction(c(1,1,0,0),
           c(1,1,0,0))

prediction() functino comes from prediction {ROCR} it has this site: http://rocr.bioinf.mpi-sb.mpg.de/

The above is a working example. As per the documentation the first parameter is 'predictions' and the second 'labels' (they would be the true values).

The output is this, which I do not fully understand, specially why there is a '2' in "fp". :

An object of class "prediction" Slot "predictions": [[1]] [1] 1 1 0 0

Slot "labels": [[1]] [1] 1 1 0 0 Levels: 0 < 1

Slot "cutoffs": [[1]] [1] Inf 1 0

Slot "fp": [[1]] [1] 0 0 2

| improve this question | |
$\endgroup$
  • $\begingroup$ Please explain what you're trying to do: is this 'prediction' function from any library, if yes which one? Also please refrain from using screenshots: meta.stackoverflow.com/a/285557/7311767 $\endgroup$ – Erwan Mar 16 at 17:48
1
$\begingroup$

The slot "fp" counts how many false positives there are at each choice of classification cutoff (which can be found in the "cutoff" slot). The cutoff represents at what value you set the threshold to binarize the numerical values into classes. Your output already appears to be binary classes, so the concept of a threshold doesn't really make much sense, but the package still tries, setting the potential thresholds at Infinity, 0, and 1. When you set the threshold at 0, everything gets classified positive, including the 2 actual negative samples - when the classification threshold is 0, you get 2 false positives.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Now I am even more confused than before. I do not understand the 'cutoff' thing. Why would the package override my prediction... To get 2 fp, predictions shall be either 1 1 1 1 or 0 0 1 1, why is predicting that?! I have passed my own values! $\endgroup$ – Chicago1988 Mar 16 at 18:04
  • $\begingroup$ @Chicago1988 The function comes from an ROC package, which is a type of analysis that only makes sense when you can vary the decision threshold (some prediction algorithms will output a continuous score that must be threshold to get a classification). Here, you have a fixed classification, so you might be better off with methods to analyze a single confusion matrix, like rdocumentation.org/packages/crossval/versions/1.0.3/topics/…. ROC methods will examine all possible confusion matrices under all possible cutoffs, including all positive/all negative predictions. $\endgroup$ – Nuclear Wang Mar 16 at 18:10
  • 1
    $\begingroup$ @Chicago1988 As a concrete example, suppose your algorithm makes 3 continuous-score predictions of {0.1, 0.4, 0.9}. You can set the threshold at 0 and call all your predictions positive, set it at 0.1 and get 1 neg and 2 pos; set it at 0.4 and get 2 neg and 1 pos, or set it at 0.9 and get all 3 neg. This ROC package treats your predictions as continuous values and tries to set thresholds, since it doesn't know the values are already binarized. What you get in this case isn't very useful, as you get either your original classification, or the degenerate cases of all pos/all neg. $\endgroup$ – Nuclear Wang Mar 16 at 18:18
  • $\begingroup$ Lastly... what threshold gets chosen? the one that maximizes my results? $\endgroup$ – Chicago1988 Mar 16 at 18:58
  • 1
    $\begingroup$ @Chicago1988 ROC analysis summarizes classifier performance over all possible thresholds, but it's up to you to pick which threshold is best for your particular application. In some domains, you may accept false positives in order to be sure you don't have any false negatives (a medical screening test), but in others, you may accept false negatives to avoid false positives (recommendation systems). Generally, you want good sensitivity/specificity together, but you may value one over the other. You may pick a different threshold for the same classifier for different applications. $\endgroup$ – Nuclear Wang Mar 16 at 19:05

Not the answer you're looking for? Browse other questions tagged or ask your own question.