1
$\begingroup$

Is there any way to determine upfront how accurate a model should be before it's worth putting into production? Just aiming for the highest accuracy possible doesn't tell us anything about the money a company would save with a 75% versus 80% model.

Say we needed to predict the number of mechanical failures. If the national average is 100/year but our company is getting 125, how accurate does our model need to be (how many correct predictions does it need to make) to get our company back down the national average?

Improve this question
$\endgroup$
3
  • $\begingroup$ Pricing the costs/benefits of implementing a predictive model into a production pipeline is too broad a subject to answer here. Accuracy of the model is only one factor of many to consider. To answer your more specific question about mechanical failures, you need to describe how the model fits into business process so that it influences the rate of observed failure. $\endgroup$
    – Neil Slater
    Mar 25, 2017 at 13:55
  • $\begingroup$ I figured the same. Wanted to see if anyone had a basic approach they used to ballpark a target accuracy. Thank you. $\endgroup$
    – Cybernetic
    Mar 25, 2017 at 14:03
  • $\begingroup$ Is machine failures your actual variable of interest? If so, then final accuracy of the model may not be as important as correctly identifying factors you can influence to improve the life of the machine part. That is, an unimpressive model might suit your purpose very well if it identified a machine setting or part attribute you could change to decrease the mechanical failure rate. $\endgroup$
    – Robert de Graaf
    Mar 27, 2017 at 1:07

1 Answer 1

Reset to default
4
$\begingroup$

The accuracy level that should satisfy you depends on the reliability that your customer would have on it and the costs matrix of false predictions.

He should put into consideration the costs of the unpredicted failures (low recall of your model) causing him production delays and the costs of over/false maintenance alarms (low precision of your model). If you know these costs you can minimize the expected cost equation which considers the precision recall and accuracy of your model:

Expected cost = p(true positive) × "maintenance cost" + p(false positive) × "inspection cost" + p(true negative) × 0 (no action) + p(false negative) × "mechanical failure cost"

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.