I am wondering what is the best way to calculate the uncertainty for my categorical predictions. I have created a model that predicts what rating a movie is getting based on certain keywords and the amount of times these key words are mentioned. I have trained the model with a small data matrix and now I am running it though a bigger data matrix that has data from over 20,000 reviews. I have used different models and different variations of my dataset (feature engineering and feature selection) and now that I have my results from my best model/dataset combination, I want to learn how to calculate the uncertainty of my results per rating. Could I use the confusion matrix or other metrics (sensitivity etc) to calculate the uncertainty?

Example Predictions: Rating1: 1000 Rating2: 25 Rating3: 59 Rating4: 599 Rating5: 3569

Thank you!

  • 1
    $\begingroup$ What do yo mean by "uncertainty" ? Is it the average error of your predictor ? Or the variance of your predictor error (see bias-variance tradeof) ? Or the self-estimated confidence that your model has about its predictions ? $\endgroup$
    – xtof54
    Commented Dec 22, 2020 at 14:49
  • $\begingroup$ I am looking more towards the self-estimated confidence that the model has for the predictions, thanks for the follow-up $\endgroup$
    – brad987
    Commented Dec 22, 2020 at 20:41


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.