0
$\begingroup$

I am looking for a confidence of model to predict well in a given situation.

So I have a model $f$ (generic, let's exemplify with a regression model of explicit form for brevity). It well fits on the train-set (by looking at say $r^2$), it well performs on the test set (by looking on RMSE). The fit is fine enough to predict with this model. Yet I want to know how confident am I to predict in this particular case, so while typically I look at the confidence $C$ of the model:

$C(f)$

while now I want to know what is the confidence of predicting with this model (subject to $x$):

$C(y=f(x))$

and I believe this shall strongly be related to presence of similar observations in the train/test sets and performance on similar observations.

Any hints on this welcomed.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

If I understand your question correctly, then this is not possible for a "generic" model. For a generic model, you can only use generic methods. One of the most generic would be bootstrap sampling. In this scenario, you take your model and train it on different bootstrap samples and derive confidence intervals from the distribution of these predictions.

However, it sounds like you are looking for a method that does not require training. In this case, you would need to use models from which confidence intervals can be derived directly based on the model assumptions, such as linear regression, Gaussian processes or Bayesian models.

Improve this answer
$\endgroup$
2
  • $\begingroup$ thanks, so let's then replace 'generic model' to 'linear regression'. Then isn't the confidence interval anyhow constant for the model, regardless the $x$ that we predict on? $\endgroup$
    – Intelligent-Infrastructure
    Sep 3, 2018 at 8:05
  • 1
    $\begingroup$ you need the "standard error of prediction": $\hat y_0 \pm t^{\alpha/2}_{n-p}\hat\sigma \sqrt{1+x_0^T(X^TX)^{-1}x_0}$ if you want more details I suggest asking a separate question $\endgroup$
    – oW_
    Sep 4, 2018 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.