Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am new to Machine Learning.
I want to know if there is any indicator which can show us ML's confidence about any given prediction.
I am suppose to build an application in which I only want to use predictions which are 100% right...anything less than that, I want to ignore those predictions.
Please let me know.
In order to have 100% certainty about the results you shouldn't use any statistical ML method, because by definition statistical methods are meant to deal with uncertainty. In other words a statistical ML algorithm is intended to provide the most likely prediction for every possible instance, there cannot be any guarantee that the predictions are correct.
There are methods to measure confidence as well, but it's extremely rare in real-world scenarios that the confidence would be 100%.
There are symbolic methods based on formal logic which guarantee 100% correctness, but of course these methods are much more limited in scope than statistical ML. For example Prolog is a solver which finds solutions based on some predefined constraints. Nowadays these methods are not considered part of ML, although they have some applications related to AI.
Suppose you train your model on pictures of dogs and ducks. You then test the model on a test set of pictures of dogs and ducks and it classifies them all perfectly correctly, 100%.
That cannot exclude the possibility that there exists a picture of a dog not in your test set that your model will say is a duck (we'll neglect the possibility that any images in your train/test set are wrongly labelled).
Your confidence in machine learning is given by how well your model performs on the test set. If it does manage to classify all 100 of your dog/duck test pictures, you only know that the performance is >99.5%. You can't say it is 100% when tested on the population of possible pictures of dogs and ducks, past, present and yet-to-be-photographed.
Unless you have a known finite set of subjects that you've already confirmed you can classify correctly 100% of the time, then I don't see how any system can know if it is working "perfectly" when given new data.
The only way to really get perfection in this sense is to define a dog as anything your model says is a dog, and a duck is anything your model says is a duck.
I have something of an answer from a Neural Network context.
If you consider a Neural Network set up for classification with a softmax activation on it's output layer, the outputs could be interpreted as something of a confidence. Practically, your loss function during training would probably choose a level of "confidence" well below 1.0 as the network choosing something. Which does not give an incentive to choose 1.0 but maybe with the right loss function. In my experience getting fancy with loss functions it just made the network be boldly incorrect or make no strong predictions for fear of being wrong.
In a softer network (one which tries to predict a future value, not just classify it) you could maybe try to include an output neuron representing confidence which you multiply by the incorrectness of the prediction for your loss. I don't know if a network will ever converge on this function practically. I have done Erwan's answer of building another network with a softmax output classification of whether or not it thinks the first network will be wrong.
With a softmax type activation you can choose to only act on predictions which are 1.0. How often this happens and how often a perfect confidence is correct is something only one's data can only bear out.
Model confidence is the model's attempt at calculating the probability of that event being true. Use model confidence ( model.predict_proba ) as a guide but don't trust it too much either. However, whenever the model is very confident about a prediction, its MOST LIKELY true. There's a technique that uses this to improve the quality of the predictions called Pseudo-labeling which consists in using the test datapoints the model is very confident about, and use them alongside the model's predictions for those points in the training set.
