1
$\begingroup$

Consider that I have a dataset automatically acquired by a machine that returns the following measurements:

[111, 121, 114, 154, 149, 150]

I then go and manually check how these values received by the machine compare to the true values, and I get the following measurements when checking manually:

[112, 121, 114, 154, 149, 149]

As you can see, the datasets differ in two places (I measured 112 where the machine saw 111 and I measured 149 where the machine saw 150), meaning the machine is inaccurate.

With that, what's the proper way to state and calculate how confident I am in how accurate or inaccurate the machine is? I could obviously say it was wrong 2/6 times (33% inaccurate/66% accurate) but I wasn't sure if there's a better way to represent this, especially with a larger dataset than the one I've listed for the example.

As a follow-up question, how could I compare these confidence or accuracy levels? For example, if this machine is usually 94% accurate but was recently improved to be 98% accurate, how would I state that apart from it's 4% more accurate?

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ This seems like it may be a regression problem involving continuous measurements rather than a classification problem involving discrete classes. By treating this as a classification problem, you treat any misclassifications as "equally wrong". Is it really the case that a prediction of 149 for a target of 150 is no better than a prediction of 1, or a prediction of 10,000? You may want to use the concept of error rather than accuracy - not whether you hit the target exactly or not, but how far from the target your prediction is. $\endgroup$
    – Nuclear Hoagie
    Sep 1, 2021 at 20:03

1 Answer 1

Reset to default
1
$\begingroup$

EDIT I’ll leave this up for true classification tasks, but I agree with the comment that the OP does not describe a classification task.

You could phrase it in terms of error rate.

You start out with a success rate of $94\%$, so an error rate of $6\%$.

Now you have a success rate of $98\%$, so an error rate of $2\%$.

You reduced the error rate by $1-\dfrac{2}{6}\approx 67\%$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.