Share Your Experience: Take the 2024 Developer Survey

Questions tagged [confidence]

The tag has no usage guidance.

32 questions
Filter by
Sorted by
Tagged with
16 views

Calculating prediction confidence from a sequence of token-level confidences

I am working with OCSR (optical chemical structure recognition) models, and they output a sequence of token-level confidences. I am looking for a method of summarising these token-level confidences ...
1 vote
35 views

Is there a model that can predict continuous data while also providing a level of confidence in the prediction?

The problem with Bayesian neural network seems to be that it is primarily working for classification problems. Is it possible to adjust this neural network, or even use a different model if one exists,...
• 13
10 views

Computing empirical coverage in binary classification confidence estimation

I am trying to compute confidence interval empirical coverage. I have computed the confidence intervals for binary probability predictions. For example, the model estimates that there is probability ...
26 views

How to compute confidence interval xgboost regressor?

I have time series data to predict values for the next 6 months. I have an xgboost model that predicts the six individual months, for the business what is important is that the cumulative value of ...
1 vote
15 views

How to compare 55 models using AUC bootstrap confidence intervals?

I want to check if there is a difference in the confidence intervals of 55 models and select just one model. What should I do?
• 13
16 views

Understanding the code for calculating the 95% confidence interval of AUC using bootstrapping

It's really embarrassing, but I lack statistical knowledge. I would like to find the confidence interval for AUC at 95%. Actually, I got the code from here(https://stackoverflow.com/questions/52373318/...
• 13
157 views

Understand and compute confidence interval and coefficient of variation for regression model

I would like to better understand the concepts of: coefficient of variation and confidence interval. Trivially taking the definitions from wikipedia: confidence interval (CI) In frequentist ...
• 1
1 vote
26 views

Correctness of derivation for binary F1 variance for F1 confidence intervals

I'm developing a python library for confidence intervals for common accuracy metrics, with both analytic and bootstrap computations. Following this paper, I implemented the Macro and Micro F1 scores ...
• 121
1 vote
306 views

Precision vs probability

Say I have a model which predicts a class $C_i$ from an input $X$, with a probability of 0.95 i.e $P(C_i| X)=0.95$. That would mean that if we do this over and over, then 95/100 times we would be ...
• 500
1 vote
49 views

Is there a way to quantify uncertainty in classification?

I'm thinking of a way to build an extension to a binary classifier (actually I will get the output probabilities like in logistic regression, so technically you should call this regression) that ...
• 65
11 views

How to find confidence sets for a bunch of regression functions?

Unlike confidence intervals where we are interested in indicating a range of beta values in which the true parameter lies 95% of all times, I would like to understand how confidence sets are ...
• 135
24 views

Statistical significance in multiple groups

What is the most appropriate test to validate significance in the case I'm comparing multiple different groups? e.g. I have different groups of ages (teenager, young, adult, elderly) and a conversion ...
• 133
1 vote
74 views

Overall acurracy +/- E (with 90% C.I.)

I am assessing the accuracy of my classification model. I performed a 4-folds cross-validation and I obtained the following Overall Accuracy: OA = (0.910, 0.920, 0.880, 0.910). So, the average OA is 0....
• 63
1 vote
92 views

How to test the confidence for a rule based system?

I have a multi-class dataset and I generated based on it rules. That is, if certain features are seen then it must be a certain class. I chose only rules with precision 1 (with respect to the whole ...
1 vote
404 views

How to use confidence labels?

I have 2 sets of training data in csv files. The training data have class labels, 1 for memorable, and 0 for not memorable. In addition, there is also a confidence label for each sample. The class ...
1 vote
57 views

Evaluate Dendrogram Statistical Significance

I have N=21 objects and each one has about 80 possible not NaN descriptors. I carried out a hierarchical clustering on the objects and I obtained this dendrogram. I want some kind of 'confidence' ...
• 111
1 vote
72 views

How are the confidence intervals of a model interpreted?

I am doing some work with R and after obtaining the confusion matrix I have obtained the following metrics corresponding to a logistic regression: ...
• 314
1 vote
214 views

Association rules - Find 100% confidence rules [closed]

Suppose there are 100 items, numbered 1 to 100, and also 100 baskets, also numbered 1 to 100. Item i is in basket b if and only ...
• 11
1 vote
657 views

Confidence intervals for evaluation on test set

I'm wondering what the "best practise" approach is for finding confidence intervals when evaluation the performance of a classifier on the test set. As far as I can see, there are two ...
• 43
38 views

Excluding data via confidence score: Is it a good idea?

Let's say I have a model which has a binary classification task (Two classes of 0 and 1) and therefore, it outputs a number between 0 and 1, if it is greater than 0.5 we consider it to be class 1 and ...
1 vote
26 views

How to State The Confidence of Accuracy/Inaccuracy?

Consider that I have a dataset automatically acquired by a machine that returns the following measurements: [111, 121, 114, 154, 149, 150] I then go and manually ...
• 111
167 views

How to predict only those values that our model is 95% sure of?

I have 5 classes. I made a XGBoost Classification model and used model.predict(test) to predict the classes of test dataset. Out of all those values predicted by my ...
• 105
1 vote
734 views

Plotting confidence intervals

For the following dataframe, I am trying to plot the means of a sample of 5 random rows . And also plot their respective confidence intervals using errorbars. I am unable to figure how to plot the ...
104 views

Producing a confidence output to use in a weighted average layer

I am trying to solve a problem where the input consists of an unordered set of observations (variable size), and the desired output is a single value describing some property of the observations as a ...
1 vote
59 views

Confidence score for all observations is between 0.50 - 0.55

Hello Data Science Stack Exchange Community, This question will appear to be open-ended, however any answers or thought will be much appreciated. I am trying to go-through a pre-trained Random Model ...
• 257
511 views

Narrow confidence Interval of forecast

I am new to data science so please accept my apology in advance. I am trying to predict the value using ARIMA. I have got weekly value for the current year. Based on the available weekly values, I ...
• 121
1 vote
61 views

Confidence in the rewards for a RL task

For a RL task that I am trying to solve, for which I train once per day, I have the rewards stored for each of those days, so that I can see the progress on daily basis. In the beginning of the ...
1 vote
67 views

Interpreting confidence interval results for datasets

I have created a dataset automatically and wanted to clarify my interpretation of the amount of noise using the confidence interval. I selected a random sample and manually annotated the sample and ...
• 23
221 views

In classification tasks, we can interpret the output vector as how "confident" the model is that the input has a certain label. For example, ...
• 123
140 views

Relation between Cross Validation and Confidence Intervals

I've read from a source which I forgot where that 'In cross validation, the model with best scores at 95% confidence interval is picked'. But according to my stat knowledge, in order for CI (...
• 103
1 vote
36 views

Understand how to simulate a statistics [closed]

This solution describes how to simulate statistics to find a confidence interval. A journalist called 1000 people in town to ask who will they be voting for out of candidates A and B. The observed ...
• 11