3
$\begingroup$

Aleatoric uncertainty refers to the notion of randomness that there is in the outcome of an experiment that is due to inherently random effects.

Epistemic uncertainty refers to the ignorance of the decision-maker, due to for example lack of data.

Aleatoric uncertainty is irreducible, while epistemic can be mitigated (Adding more data).

When we deploy a ML model in production. Can we distinguish between epistemic and aleatoric uncertainy?

$\endgroup$
2
  • 1
    $\begingroup$ one way to quantify aleatoric uncertainty is as average uncertainty over various models, since then model mismatch will average out leaving only irreducible uncertainty $\endgroup$
    – Nikos M.
    Jul 29, 2021 at 17:36
  • $\begingroup$ Related: datascience.stackexchange.com/questions/111717/… $\endgroup$
    – Rexcirus
    Dec 29, 2023 at 12:29

2 Answers 2

1
$\begingroup$

In Bias-Variance tradeoff theorem, aleatoric uncertainty is represented by the irreducible error (inherently and irreducibly random). The rest represents model mismatch due to imprecise knowledge of the generation of the problem.

One way to quantify aleatoric uncertainty is as average uncertainty over various models for the same problem, as then uncertainty due to model mismatch will tend to average out leaving only the irreducible uncertainty.

$\endgroup$
3
  • $\begingroup$ That will be a way to measure uncertainty. But you can not say which one it is. Aggregating different models (similar than bagging) will allow you to calculate uncertainty in general. $\endgroup$ Jul 29, 2021 at 21:25
  • $\begingroup$ Well, theoretically the average over all possible models should provide the irreducible uncertainty. In practice it can at least provide an upper bound. $\endgroup$
    – Nikos M.
    Jul 29, 2021 at 22:05
  • $\begingroup$ If you dont have data about the current weather in France, and they ask you about the weather in France, even if you average several models, you will still have epistemic uncertainty. $\endgroup$ Mar 13, 2022 at 18:22
0
$\begingroup$

The general problem seems very hard. How would you (as a human) distinguish between epistemic and aleatoric uncertainy? For edge cases, you would need to push the boundaries of physics to understand it. For instance in the case of weather forecast, it is unclear how much a three-weeks forecast uncertainty estimate can be lowered with additional data.

In more mundane cases, you could add additional meta-parameters to your model to work as epistemic-vs-aleatoric classifiers. Following the example in your comments, asking about the current weather in France, your model could search for any weather data point tagged France. If none or not enough are found data-points are found, the model should reply that not enough information on France is available. The "not enough" threshold should be based on theoretical minimum amount of data to perform a reliable forecast. How reliable is up to you. In the limit you could demand enough data to bring epistemic uncertainty to zero.

$\endgroup$
1
  • $\begingroup$ If you ask me whats the weather in France, I don't know because I have no data. If you ask me the weather in France in 3 weeks I don't know because its not possible to calculate, hence aleatoric. $\endgroup$ Dec 30, 2023 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.