Questions tagged [confidence]
The confidence tag has no usage guidance.
37 questions
4
votes
1
answer
47
views
Nested-cross validation pipeline and confidence intervals
I'm hoping someone can help me think through this. I've come across a lot of different resources on nested-cv, but I think I'm confused as to how to go about model selection and the appropriate ...
0
votes
0
answers
8
views
How to judge the model's confidence in Learning-To-Rank (LTR)?
Recently I have been working on Learning-To-Rank model. I think it's very useful for my purpose, but how can we extract information regarding its confidence and accuracy? Specifically, I am using the ...
- 1
0
votes
0
answers
14
views
How to adjust classification totals based on known bias of estimator
Let's say I have a dataset, $D$, with known ground truth labels. I nonetheless use a few-shot LLM classifier on this dataset to predict $k$ classes for each label.
From the LLM results, I get ...
0
votes
0
answers
8
views
What's a suitable error metric for quantifying uncertainty in future projections using a model ensemble?
Rather than just providing a mean projection, I'm looking to provide a likely range of projections using output from 9 models.
Each dataset consists of spatial maximum probability values [0,1] for a ...
- 101
0
votes
0
answers
31
views
Estimate confidence interval of a cubic B-spline fit
I have a non-linear system. I apply multiple inputs x(t) (t=time) and measure each response y(t).
In other words, I have inputs x1(t), x2(t), x3(t), xn(t) and I measure y1(t), y2(t), y3(t), yn(t) for ...
- 1
0
votes
0
answers
26
views
Calculating prediction confidence from a sequence of token-level confidences
I am working with OCSR (optical chemical structure recognition) models, and they output a sequence of token-level confidences. I am looking for a method of summarising these token-level confidences ...
- 101
2
votes
1
answer
106
views
Is there a model that can predict continuous data while also providing a level of confidence in the prediction?
The problem with Bayesian neural network seems to be that it is primarily working for classification problems. Is it possible to adjust this neural network, or even use a different model if one exists,...
- 23
0
votes
0
answers
22
views
Computing empirical coverage in binary classification confidence estimation
I am trying to compute confidence interval empirical coverage. I have computed the confidence intervals for binary probability predictions. For example, the model estimates that there is probability ...
0
votes
0
answers
233
views
How to compute confidence interval xgboost regressor?
I have time series data to predict values for the next 6 months.
I have an xgboost model that predicts the six individual months, for the business what is important is that the cumulative value of ...
1
vote
0
answers
23
views
How to compare 55 models using AUC bootstrap confidence intervals?
I want to check if there is a difference in the confidence intervals of 55 models and select just one model. What should I do?
- 13
0
votes
0
answers
92
views
Understanding the code for calculating the 95% confidence interval of AUC using bootstrapping
It's really embarrassing, but I lack statistical knowledge. I would like to find the confidence interval for AUC at 95%. Actually, I got the code from here(https://stackoverflow.com/questions/52373318/...
- 13
0
votes
1
answer
194
views
Understand and compute confidence interval and coefficient of variation for regression model
I would like to better understand the concepts of:
coefficient of variation and confidence interval.
Trivially taking the definitions from wikipedia:
confidence interval (CI)
In frequentist ...
- 1
1
vote
0
answers
30
views
Correctness of derivation for binary F1 variance for F1 confidence intervals
I'm developing a python library for confidence intervals for common accuracy metrics, with both analytic and bootstrap computations.
Following this paper, I implemented the Macro and Micro F1 scores ...
- 121
1
vote
1
answer
456
views
Precision vs probability
Say I have a model which predicts a class $C_i$ from an input $X$, with a probability of 0.95 i.e $P(C_i| X)=0.95$. That would mean that if we do this over and over, then 95/100 times we would be ...
- 510
1
vote
0
answers
49
views
Is there a way to quantify uncertainty in classification?
I'm thinking of a way to build an extension to a binary classifier (actually I will get the output probabilities like in logistic regression, so technically you should call this regression) that ...
- 75
0
votes
0
answers
11
views
How to find confidence sets for a bunch of regression functions?
Unlike confidence intervals where we are interested in indicating a range of beta values in which the true parameter lies 95% of all times, I would like to understand how confidence sets are ...
- 135
0
votes
1
answer
28
views
Statistical significance in multiple groups
What is the most appropriate test to validate significance in the case I'm comparing multiple different groups? e.g.
I have different groups of ages (teenager, young, adult, elderly) and a conversion ...
- 133
1
vote
1
answer
77
views
Overall acurracy +/- E (with 90% C.I.)
I am assessing the accuracy of my classification model. I performed a 4-folds cross-validation and I obtained the following Overall Accuracy: OA = (0.910, 0.920, 0.880, 0.910). So, the average OA is 0....
- 63
1
vote
1
answer
113
views
How to test the confidence for a rule based system?
I have a multi-class dataset and I generated based on it rules. That is, if certain features are seen then it must be a certain class. I chose only rules with precision 1 (with respect to the whole ...
- 131
1
vote
1
answer
440
views
How to use confidence labels?
I have 2 sets of training data in csv files. The training data have class labels, 1 for memorable, and 0 for not memorable. In addition, there is also a confidence label for each sample. The class ...
user135529
1
vote
0
answers
64
views
Evaluate Dendrogram Statistical Significance
I have N=21 objects and each one has about 80 possible not NaN descriptors.
I carried out a hierarchical clustering on the objects and I obtained this dendrogram.
I want some kind of 'confidence' ...
- 111
1
vote
2
answers
81
views
How are the confidence intervals of a model interpreted?
I am doing some work with R and after obtaining the confusion matrix I have obtained the following metrics corresponding to a logistic regression:
...
- 324
1
vote
0
answers
234
views
Association rules - Find 100% confidence rules [closed]
Suppose there are 100 items, numbered 1 to 100, and also 100 baskets, also numbered 1 to 100. Item i is in basket b if and only ...
- 11
1
vote
1
answer
776
views
Confidence intervals for evaluation on test set
I'm wondering what the "best practise" approach is for finding confidence intervals when evaluation the performance of a classifier on the test set.
As far as I can see, there are two ...
- 43
0
votes
1
answer
39
views
Excluding data via confidence score: Is it a good idea?
Let's say I have a model which has a binary classification task (Two classes of 0 and 1) and therefore, it outputs a number between 0 and 1, if it is greater than 0.5 we consider it to be class 1 and ...
1
vote
1
answer
26
views
How to State The Confidence of Accuracy/Inaccuracy?
Consider that I have a dataset automatically acquired by a machine that returns the following measurements:
[111, 121, 114, 154, 149, 150]
I then go and manually ...
- 111
0
votes
1
answer
223
views
How to predict only those values that our model is 95% sure of?
I have 5 classes. I made a XGBoost Classification model and used model.predict(test) to predict the classes of test dataset. Out of all those values predicted by my ...
- 105
1
vote
1
answer
855
views
Plotting confidence intervals
For the following dataframe, I am trying to plot the means of a sample of 5 random rows . And also plot their respective confidence intervals using errorbars. I am unable to figure how to plot the ...
- 143
0
votes
1
answer
109
views
Producing a confidence output to use in a weighted average layer
I am trying to solve a problem where the input consists of an unordered set of observations (variable size), and the desired output is a single value describing some property of the observations as a ...
1
vote
1
answer
113
views
Confidence score for all observations is between 0.50 - 0.55
Hello Data Science Stack Exchange Community,
This question will appear to be open-ended, however any answers or thought will be much appreciated. I am trying to go-through a pre-trained Random Model ...
- 257
0
votes
0
answers
558
views
Narrow confidence Interval of forecast
I am new to data science so please accept my apology in advance. I am trying to predict the value using ARIMA. I have got weekly value for the current year. Based on the available weekly values, I ...
- 121
1
vote
0
answers
64
views
Confidence in the rewards for a RL task
For a RL task that I am trying to solve, for which I train once per day, I have the rewards stored for each of those days, so that I can see the progress on daily basis.
In the beginning of the ...
- 11
1
vote
1
answer
76
views
Interpreting confidence interval results for datasets
I have created a dataset automatically and wanted to clarify my interpretation of the amount of noise using the confidence interval.
I selected a random sample and manually annotated the sample and ...
- 23
2
votes
2
answers
275
views
Confidence rating for regression tasks
In classification tasks, we can interpret the output vector as how "confident" the model is that the input has a certain label. For example,
...
- 123
2
votes
1
answer
146
views
Relation between Cross Validation and Confidence Intervals
I've read from a source which I forgot where that 'In cross validation, the model with best scores at 95% confidence interval is picked'.
But according to my stat knowledge, in order for CI (...
- 103
1
vote
1
answer
36
views
Understand how to simulate a statistics [closed]
This solution describes how to simulate statistics to find a confidence interval.
A journalist called 1000 people in town to ask who will they be voting for out of candidates A and B.
The observed ...
- 11
2
votes
2
answers
5k
views
Predict_proba for Binary classifier in Tensorflow
I'm working in binary classifier problem, where I have used Tensorflow low level API's.
Last layer wrapped with Sigmoidal Function and just returning a single value.
For my prediction I just set a ...
- 1,282