Questions tagged [confidence]
The confidence tag has no usage guidance.
17 questions with no upvoted or accepted answers
1
vote
0
answers
23
views
How to compare 55 models using AUC bootstrap confidence intervals?
I want to check if there is a difference in the confidence intervals of 55 models and select just one model. What should I do?
- 13
1
vote
0
answers
30
views
Correctness of derivation for binary F1 variance for F1 confidence intervals
I'm developing a python library for confidence intervals for common accuracy metrics, with both analytic and bootstrap computations.
Following this paper, I implemented the Macro and Micro F1 scores ...
- 121
1
vote
0
answers
49
views
Is there a way to quantify uncertainty in classification?
I'm thinking of a way to build an extension to a binary classifier (actually I will get the output probabilities like in logistic regression, so technically you should call this regression) that ...
- 75
1
vote
0
answers
64
views
Evaluate Dendrogram Statistical Significance
I have N=21 objects and each one has about 80 possible not NaN descriptors.
I carried out a hierarchical clustering on the objects and I obtained this dendrogram.
I want some kind of 'confidence' ...
- 111
1
vote
1
answer
856
views
Plotting confidence intervals
For the following dataframe, I am trying to plot the means of a sample of 5 random rows . And also plot their respective confidence intervals using errorbars. I am unable to figure how to plot the ...
- 143
1
vote
0
answers
64
views
Confidence in the rewards for a RL task
For a RL task that I am trying to solve, for which I train once per day, I have the rewards stored for each of those days, so that I can see the progress on daily basis.
In the beginning of the ...
- 11
0
votes
0
answers
8
views
How to judge the model's confidence in Learning-To-Rank (LTR)?
Recently I have been working on Learning-To-Rank model. I think it's very useful for my purpose, but how can we extract information regarding its confidence and accuracy? Specifically, I am using the ...
- 1
0
votes
0
answers
14
views
How to adjust classification totals based on known bias of estimator
Let's say I have a dataset, $D$, with known ground truth labels. I nonetheless use a few-shot LLM classifier on this dataset to predict $k$ classes for each label.
From the LLM results, I get ...
0
votes
0
answers
8
views
What's a suitable error metric for quantifying uncertainty in future projections using a model ensemble?
Rather than just providing a mean projection, I'm looking to provide a likely range of projections using output from 9 models.
Each dataset consists of spatial maximum probability values [0,1] for a ...
- 101
0
votes
0
answers
33
views
Estimate confidence interval of a cubic B-spline fit
I have a non-linear system. I apply multiple inputs x(t) (t=time) and measure each response y(t).
In other words, I have inputs x1(t), x2(t), x3(t), xn(t) and I measure y1(t), y2(t), y3(t), yn(t) for ...
- 1
0
votes
0
answers
26
views
Calculating prediction confidence from a sequence of token-level confidences
I am working with OCSR (optical chemical structure recognition) models, and they output a sequence of token-level confidences. I am looking for a method of summarising these token-level confidences ...
- 101
0
votes
0
answers
22
views
Computing empirical coverage in binary classification confidence estimation
I am trying to compute confidence interval empirical coverage. I have computed the confidence intervals for binary probability predictions. For example, the model estimates that there is probability ...
0
votes
0
answers
236
views
How to compute confidence interval xgboost regressor?
I have time series data to predict values for the next 6 months.
I have an xgboost model that predicts the six individual months, for the business what is important is that the cumulative value of ...
0
votes
0
answers
92
views
Understanding the code for calculating the 95% confidence interval of AUC using bootstrapping
It's really embarrassing, but I lack statistical knowledge. I would like to find the confidence interval for AUC at 95%. Actually, I got the code from here(https://stackoverflow.com/questions/52373318/...
- 13
0
votes
0
answers
11
views
How to find confidence sets for a bunch of regression functions?
Unlike confidence intervals where we are interested in indicating a range of beta values in which the true parameter lies 95% of all times, I would like to understand how confidence sets are ...
- 135
0
votes
1
answer
28
views
Statistical significance in multiple groups
What is the most appropriate test to validate significance in the case I'm comparing multiple different groups? e.g.
I have different groups of ages (teenager, young, adult, elderly) and a conversion ...
- 133
0
votes
0
answers
558
views
Narrow confidence Interval of forecast
I am new to data science so please accept my apology in advance. I am trying to predict the value using ARIMA. I have got weekly value for the current year. Based on the available weekly values, I ...
- 121