1
$\begingroup$

I've seen lots of tutorials and papers about this or that model getting some great accuracy score. In this case, let's say 85%. But what I never see is what you are supposed to do with the remaining 15%? I'm guessing that most of these people I'm reading are academics or scientists trying to make a point, but what happens when you are in a commercial industry where 85% just won't cut it? What do you do with that 15%? Do you need more epochs? A different model? A different architecture? What? Thanks.

$\endgroup$

3 Answers 3

0
$\begingroup$

This is where so-called term Baseline comes into play. One needs to have a baseline either a simple model prediction performance (accuracy, precision or recall whatever) set, and try to improve upon it. Or in a more natural way, when available, it is best to have a human baseline. The latter is quite common in industries, but it is more costly to obtain since for a collection of data human labels are needed.

Where to go with the reaming failed % of prediction, it various. Often many take a model-centric approach, since it sounds the most obvious thing to do with many new mode, architecture being published and released, and improve further. Often though, that doesn't help much and it is usually time-consuming. Others may take a data-centric approach. Meaning they would focus more on data, either data has the right quality, labels and annotations are correct, and most importantly whether it is enough considering the model that is being used to train on. For example, Deep Neural Networks are quite greedy. That means the more data the better. Or even more may ask more fundamental questions that whether such data is a good representation for machine judgement (rule of thumb here is if human is able to make a judgement under X secs/msec, machine can do too).

There are also others that consider both approach and take the best from each world. Have they eyes wide open on the data, yet doing fair amount on the model part either to be hyper-parameter optimization, or choice of mode or model's architecture etc.

At the end of the day. One needs to ask himself or herself, what if there was no model. Are you benefiting of model predictions even at 85% accuracy? Or the costs of doing wrong prediction is so that, only having 98% recall would be acceptable, otherwise one is better off with manual processes, till it gets there. Here is when the business expectations are vital, and come hand in hand with ML/DS practitioners.

I strongly recommend watching Andrew Ng's Recent Talk on From Model-centric to Data-centric AI. He gives practical examples of such scenarios, would you more insights into the matter.

$\endgroup$
2
  • 1
    $\begingroup$ I have lecture notes covering many useful techniques for improving a model by improving its data. These come from my course "Introduction to Data-Centric AI" recently taught at MIT, all materials are freely available! $\endgroup$ Apr 13, 2023 at 20:53
  • 1
    $\begingroup$ Nice, thanks for the recommendation and the link! $\endgroup$ Apr 14, 2023 at 7:13
0
$\begingroup$

Your options are:

  1. Do nothing with the model. (Then why did you build it?)

  2. Use the model, knowing that it works well enough, even though it makes mistakes.

(Evaluating if it works well enough is a separate discussion.)

Speech recognition software like Siri make mistakes, yet the software remains useful. The alternative is not to have any speech recognition at all. I would be happy to see fewer mistakes, but I find speech recognition software to be a nice convenience.

Remember that you do not know which $15\%$ of the cases out in the wild are the failures (until you get feedback from a user, which might be never).

$\endgroup$
0
$\begingroup$

In short I would say:

  • stick with the old probably manual process if it suits you better
  • or try to understand why the result is only 85% and improve the model/data quality
  • try some other models, parameters, engineer other features
  • get more and cleaner data
  • or hire a top DataScientist who could help with that issue

When deciding if 85% is enough or not, you can compare it with the success percentage of the old process and/or calculate and compate the cost of the old and ML 85% process to help decide which is best for you or could justify investing time or money improving it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.