5
$\begingroup$

I am creating decision trees modeling data that looks like this.

pelvic_radius   degree_spondylolisthesis    class   
82.45603817    41.6854736      Abnormal    
114.365845     -0.421010392    Normal

When finished, I run my test data through my tree and compare the outputs from the run to the ones I am given. This will allow me to check my accuracy, precision, and recall values.

TP = 0; % True Positives
TN = 0; % True Negatives
FP = 0; % False Positives
FN = 0; % False Negatives

And then once those values are calculated, I can calculate the following.

precision = TP/(TP+FP);
accuracy = (TP+TN)/(TP+TN+FP+FN);
recall = TP/(TP+FN);

However, this can be done in two ways. One considering the 'Normal' class as Positive and one Considering the 'Abnormal' class as positive.

Here is the sudo code to further explain what I mean.

for k=1:length(resultsOfTestSet)
    if(strcmp(resultsOfTestSet{k},'Normal'))
        if (strcmp(testSet{k}, 'Normal'))
            % TRUE POSITIVE
            TP = TP + 1;
        else
            % FALSE POSITIVE
            FP = FP + 1;
        end
    elseif(strcmp(resultsOfTestSet{k},'Abnormal'))
        if(strcmp(testSet{k},'Abnormal'))
            % TRUE NEGATIVE
            TN = TN + 1;
        else
            % FALSE NEGATIVE
            FN = FN + 1;
        end
    end
end

The above case assumes Normal as the 'Positive' resultant class. However by just flipping the compare statements, I can get alternate values.

for k=1:length(resultsOfTestSet)
    if(strcmp(resultsOfTestSet{k},'Abnormal'))
        if (strcmp(testSet{k}, 'Abnormal'))
            % TRUE POSITIVE
            TP = TP + 1;
        else
            % FALSE POSITIVE
            FP = FP + 1;
        end
    elseif(strcmp(resultsOfTestSet{k},'Normal'))
        if(strcmp(testSet{k},'Normal'))
            % TRUE NEGATIVE
            TN = TN + 1;
        else
            % FALSE NEGATIVE
            FN = FN + 1;
        end
    end
end

So after running it for both cases, I get the following values.

For Abnormal being my Positive case

precision = 96.5517
accur = 95
recall = 87.5000

For Normal = Positive case

precision =  94.3662
accur = 95
recall = 98.5294

So how do I calculate the combined result, prec, accu, and recall? Or, am I just missing the point and you just calculate it for one class at a time, like the one you are focusing on.

The reason I am asking is because now lets say I have a set with multiple class outcomes in my decision tree. This is where I realized I have to pick a class to determine as my positive, or just look at classes individually.

Here is a similar set with 3 class possibilities. Again, how do I calculate for the whole data set? Or is an individual class thing? Or do you calculate the individuals and then come together with a total for the whole decision tree.

pelvic_radius   degree_spondylolisthesis    class   
82.45603817    41.6854736      Abnormal    
114.365845     -0.421010392    Normal
95             25              Perfect
$\endgroup$
3
$\begingroup$

The metrics you calculate are of two types, metrics that depict the entire prediction model you have built like accuracy which will be same in both the cases of your pseudo code. While the others like precision says how precise are you in explaining particular class of interest (accuracy can also be expressed this way in multi-class classification, see the diagram). This score depends on which class you had selected as a positive one. If you put positive class as face of your model, then it is called ppv or precision and npv, if vice versa.

Coming to the multi-class classification, the core definition holds the same. Now the matrix will be n x n(n being number of classes). The sample matrix looks likes this.

enter image description here

The diagonal elements explains the number of 1 class's predicted as 1. Now there are n precision values for each class. Precision for class 1 is how many values were truly predicted as 1 divided by how many are predicted as 1 (FP's included), which is the sum of the first column.

Not but not the least if you adamantly want a precision like metrics for the entire model, you got micro & macro averaging methods, which are helpful in giving a combined metric. This blog post explains it pretty well. Hope this clears something.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Excellent, so recall and precision are class based metrics. While accuracy can both be a metric for all the classes, or determine individual accuracy of each class. $\endgroup$ – doedotdev Jan 28 '18 at 15:06
  • 1
    $\begingroup$ Yes, precisely. But as mentioned at the end, there can be a precision for a model too. $\endgroup$ – Kiritee Gak Jan 28 '18 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.