8
$\begingroup$

How to chain statistical methods (estimators or classifiers) taking into account the uncertainty (error) of the previous step?

Ex: Consider a pipeline, where housing prices are estimated from census and geographical data and are fed into another algorithm to estimate credit scores. How does the error in estimating housing prices fed into credit scores estimator and factor in the overall error?

I think that if you just consider the output values and not the error of the previous step, the error of the current estimator will be lesser and will be misleading. The uncertainty is not being propagated forward in this pipeline, hence the uncertainty at the end is only because of the last step.

I'm new to Machine Learning and in the introductory books or courses I didn't come across any discussion about this topic. If anyone can point me to good resources to learn more about this, I'll be happy.

$\endgroup$
3
+25
$\begingroup$

One option is to use Bayesian methods in machine learning.

Assume the input to any stage in a machine learning pipeline is a prior distribution and the output of any stage is a posterior distribution. Then it would be possible to chain stages together with uncertainty propagated through the system through the distributions.

$\endgroup$
2
$\begingroup$

There are three methods I can think of, two of them are dependent on the method you use:

  • If a linear model is used - the linear regression theory provides estimates for the uncertainty of the predictions.
  • If a random forest or bagging algorithm is used - we can provide an estimate of the uncertainty of the prediction by taking all the trees predictions and aggregating them using the standard deviation.
  • There's also a method to estimate errors that doesn't depend on the algorithm that is used. The idea is to train a regular model to predict the average outcome. After this model is fit, you apply the model on a new validation set, and compute the absolute errors that the model is making. After that, you use another model to predict these absolute errors. Using this second model allows you to give uncertainty estimates - if the prediction made by this second model is low, the uncertainty will be low, otherwise the uncertainty will be high.

And, of course, a Bayesian framework can be used, which is the natural language to model uncertainty in the first place.

$\endgroup$
2
  • 3
    $\begingroup$ I'm curious - Is there a established name for option #3? $\endgroup$ – Brian Spiering Oct 13 '20 at 16:28
  • 1
    $\begingroup$ No idea, I read it on a twitter thread, I am not sure if it's used a lot but it made sense to me $\endgroup$ – David Masip Oct 13 '20 at 18:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.