I am wondering what is the best way to calculate the uncertainty for my categorical predictions. I have created a model that predicts what rating a movie is getting based on certain keywords and the amount of times these key words are mentioned. I have trained the model with a small data matrix and now I am running it though a bigger data matrix that has data from over 20,000 reviews. I have used different models and different variations of my dataset (feature engineering and feature selection) and now that I have my results from my best model/dataset combination, I want to learn how to calculate the uncertainty of my results per rating. Could I use the confusion matrix or other metrics (sensitivity etc) to calculate the uncertainty?

Example Predictions: Rating1: 1000 Rating2: 25 Rating3: 59 Rating4: 599 Rating5: 3569

Thank you!

  • 1
    $\begingroup$ What do yo mean by "uncertainty" ? Is it the average error of your predictor ? Or the variance of your predictor error (see bias-variance tradeof) ? Or the self-estimated confidence that your model has about its predictions ? $\endgroup$ – xtof54 Dec 22 '20 at 14:49
  • $\begingroup$ I am looking more towards the self-estimated confidence that the model has for the predictions, thanks for the follow-up $\endgroup$ – brad987 Dec 22 '20 at 20:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.