3
$\begingroup$

I am new to Machine Learning.

I want to know if there is any indicator which can show us ML's confidence about any given prediction.

I am suppose to build an application in which I only want to use predictions which are 100% right...anything less than that, I want to ignore those predictions.

Please let me know.

$\endgroup$
2
  • 4
    $\begingroup$ This is easy! You just ignore all of your predictions, since all of them have some uncertainty. If this does not suit your needs (and I anticipate it does not), perhaps you can refine the question. $\endgroup$
    – Dave
    Jan 31, 2022 at 15:25
  • 3
    $\begingroup$ I feel a lot of the answers here get sidetracked by the 100% part, without answering what is I think the more interesting question - namely whether / which ML models can provide a measure of confidence along with the prediction (even under the fairly obvious caveat that this confidence can never be "100%" in a real, non-trivial case). $\endgroup$
    – xLeitix
    Jan 31, 2022 at 16:38

4 Answers 4

8
$\begingroup$

In order to have 100% certainty about the results you shouldn't use any statistical ML method, because by definition statistical methods are meant to deal with uncertainty. In other words a statistical ML algorithm is intended to provide the most likely prediction for every possible instance, there cannot be any guarantee that the predictions are correct.

There are methods to measure confidence as well, but it's extremely rare in real-world scenarios that the confidence would be 100%.

There are symbolic methods based on formal logic which guarantee 100% correctness, but of course these methods are much more limited in scope than statistical ML. For example Prolog is a solver which finds solutions based on some predefined constraints. Nowadays these methods are not considered part of ML, although they have some applications related to AI.

$\endgroup$
3
  • $\begingroup$ thanks for your answer. Can you tell me what methods are there to measure confidence..at least I can compare relatively between two predictions and decide on which one to consider. $\endgroup$
    – meetpd
    Jan 31, 2022 at 10:18
  • $\begingroup$ @meetpd I'm not very knowledgeable about these so don't hesitate to ask a new question here on DSSE or statsSE for more detail. As far as I know general confidence methods are experimental: take a large sample of labelled predictions, resampling many times and then measure how many predictions are correct. $\endgroup$
    – Erwan
    Jan 31, 2022 at 10:49
  • $\begingroup$ There's also quality estimation for some specific tasks (example), but this is done by training a second model which predicts the quality/confidence, so the question becomes how sure we are about the confidence predictions of this other system. These methods are all based on statistics, so none of them would guarantee that a prediction is 100% correct. The question of comparing predictions for the same instance means using multiple models, this is the area of ensemble learning. $\endgroup$
    – Erwan
    Jan 31, 2022 at 10:53
2
$\begingroup$

Suppose you train your model on pictures of dogs and ducks. You then test the model on a test set of pictures of dogs and ducks and it classifies them all perfectly correctly, 100%.

That cannot exclude the possibility that there exists a picture of a dog not in your test set that your model will say is a duck (we'll neglect the possibility that any images in your train/test set are wrongly labelled).

Your confidence in machine learning is given by how well your model performs on the test set. If it does manage to classify all 100 of your dog/duck test pictures, you only know that the performance is >99.5%. You can't say it is 100% when tested on the population of possible pictures of dogs and ducks, past, present and yet-to-be-photographed.

Unless you have a known finite set of subjects that you've already confirmed you can classify correctly 100% of the time, then I don't see how any system can know if it is working "perfectly" when given new data.

The only way to really get perfection in this sense is to define a dog as anything your model says is a dog, and a duck is anything your model says is a duck.

$\endgroup$
0
$\begingroup$

I have something of an answer from a Neural Network context.

If you consider a Neural Network set up for classification with a softmax activation on it's output layer, the outputs could be interpreted as something of a confidence. Practically, your loss function during training would probably choose a level of "confidence" well below 1.0 as the network choosing something. Which does not give an incentive to choose 1.0 but maybe with the right loss function. In my experience getting fancy with loss functions it just made the network be boldly incorrect or make no strong predictions for fear of being wrong.

In a softer network (one which tries to predict a future value, not just classify it) you could maybe try to include an output neuron representing confidence which you multiply by the incorrectness of the prediction for your loss. I don't know if a network will ever converge on this function practically. I have done Erwan's answer of building another network with a softmax output classification of whether or not it thinks the first network will be wrong.

With a softmax type activation you can choose to only act on predictions which are 1.0. How often this happens and how often a perfect confidence is correct is something only one's data can only bear out.

$\endgroup$
0
$\begingroup$

Model confidence is the model's attempt at calculating the probability of that event being true. Use model confidence ( model.predict_proba ) as a guide but don't trust it too much either. However, whenever the model is very confident about a prediction, its MOST LIKELY true. There's a technique that uses this to improve the quality of the predictions called Pseudo-labeling which consists in using the test datapoints the model is very confident about, and use them alongside the model's predictions for those points in the training set.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.