2
$\begingroup$

I have a machine learning model that try to fingerprint the functions in a binary file with a corpus. Final output of upon inputing a binary file is a table with one to one mapping between the binary function and the corpus function as follows-: enter image description here

As you could see from the names, some of the functions are correct while the others are incorrect. Is there a way to calculate precision and recall for the above result? I understand that precision and recall make sense if I am doing other ML tasks such as image classification. Using a confusion matrix will help to calculate both the metrics easily. However, I am confused and feel that I could not do such measures as it is just one to one mapping which is either true or false. If precision and recall does not make sense, is there any other metrics I could use to evaluate the model? Thank you!

Improve this question
$\endgroup$
1
  • $\begingroup$ How are these decimals values calculated? Please try to explain the underlying logic $\endgroup$
    – 10xAI
    Jul 13, 2020 at 14:23

1 Answer 1

Reset to default
5
$\begingroup$

First of all, precision and recall are not specific to image classification; they are relevant wherever there are two distinct "positive" and "negative" classes (for example, when you test an e-mail for "spam/not-spam", or a blood sample for "has virus/does not have virus").
You can read more on this question on Cross Validated, but to sum it up - precision is the probability that a sample is positive if a test said it is, and recall is the probability that a positive sample will be reported as positive by the test.
False positives mess up your precision, and false negatives mess up your recall.

Now, your task appears to be one of multi-class classification - with at least 17 classes, from your example. I wouldn't go with precision/recall for this - you can only do it pair-wise for pairs of classes. You can, however, plot a CxC confusion matrix (where C is the number of classes), and investigate where your models tend to miss. There's an implementation in SKLearn (link).

If you need a single-number metric, I'd start with just accuracy (and develop from there). Following Nuclear Wang's comment, I'd also suggest looking at Cohen's Kappa (see explanation on Cross Validated) to better account for class imbalance.

To read more on multi-class classification, see this question. I'd also recommend this blog post on Towards Data Science

Improve this answer
$\endgroup$
4
  • $\begingroup$ This is not a multi class classification problem. The thing is there is only one match function from the corpus against the binary function. Therefore, every function has it's own match. $\endgroup$
    – hEShaN
    Jul 13, 2020 at 17:05
  • 1
    $\begingroup$ That does not prevent it from being a multiclass classification problem. There are multiple classes (i.e. functions in the right side), and you want to predict to which class each sample (i.e. function in the left side) belongs. $\endgroup$
    – Itamar Mushkin
    Jul 13, 2020 at 18:03
  • 2
    $\begingroup$ @hEShaN This does appear to be a multiclass problem, since every input sample can be classified with one of many outputs - it is not a binary classification problem, since there are not only 2 choice of match function. Be a little careful with accuracy as a measure, since it's sensitive to overall class abundance - it has its place for letting you compare among models for the same data, but the absolute accuracy level will depend greatly on the underlying class proportions. The kappa statistic is a good choice of metric which accounts for matches that occur by chance. $\endgroup$
    – Nuclear Hoagie
    Jul 13, 2020 at 18:54
  • $\begingroup$ @NuclearWang that's right about Cohen's Kappa. Added it to post. $\endgroup$
    – Itamar Mushkin
    Jul 14, 2020 at 6:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.