Questions tagged [statistics]
Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.
1,107
questions
0
votes
0
answers
4
views
how do i compute the predictive covariance matrix from the posterior samples?
I have generated with EMCEE some posterior samples from a statistical model whose likelihood is a multivariate gaussian. it's a regression problem.
can you explain me how I can use these samples to ...
- 101
0
votes
0
answers
12
views
How to fit datasets accounting for standard deviation in Python?
I have multiple datasets of measurements with standard deviation and I would like to fit all the data (with a non-linear regression model) accounting for SD or SEM (it is quite similar to let the ...
0
votes
1
answer
21
views
Statistical test for comparing number of clusters in data
I am performing $K$-means clustering on a dataset consisting of $n$ observations and $d$ variables, and I'm trying to determine the optimal number of clusters. Is there a test that can determine the ...
- 121
0
votes
0
answers
15
views
Getting probability to finish in the last 3 ones after each game week
I'am working on a dataframe with five differents features :
team
game_week
season
cum_points : pre game cumulative number of points
final_position
Those datas are covering 10 seasons of Premier ...
0
votes
0
answers
20
views
Which statistical test is suitable for analyze the correlation between non-independent categorical variables?
Im working on an EDA on chocolate bar ratings, trying to indentify relationships between variables.
In this case, im trying to test if the ingredients of a chocolate bar area associated with its ...
0
votes
1
answer
14
views
How a Random forest "learns" or How loss (objective function value) is propagated back so that a random forest can "Improve"?
Every Blog and Youtube video talks about the same steps:
Choose that you have to build N number of tree and do the task 2-5 ...
- 323
0
votes
0
answers
18
views
Probability of a Maximum in a Time Series Given Past Data
I'm trying to predict the peak power usage of an EV charging station. I would like figure out probability bounds given the peak power throughout the month.
Imagine that our EV time series consists of ...
- 1
0
votes
0
answers
17
views
Scaling nominal vars K means
I had a discussion recently with a coworker. We are running a K means clustering algorithm. He said that when dummy variables are made into 1s and 0s,these columns must be scaled in a specific way. If ...
- 1
1
vote
1
answer
85
views
Trying to extrapolate info from a partial data set - statistical inference
I am wondering if my logic is OK here or not.
98% of a group without a device has an event occur
2% of groups with devices have an event occur.
Since we know that correlation isn't causation, I can't ...
- 63
1
vote
2
answers
36
views
For standard deviation's formula, why does division by sample size come before square rooting?
Why is this used to calculate a sample's standard deviation:
$s=\sqrt{\frac{1}{N-1}\sum_{i=1}^N(x_i-\bar{x})^2}$
and not something like:
$s=\frac{1}{N-1}\sqrt{\sum_{i=1}^N(x_i-\bar{x})^2}$
I ...
- 11
0
votes
1
answer
11
views
What's the word for aggregating by two attributes together, rather than separately?
Let's imagine we have this underlying data:
...
- 111
0
votes
1
answer
12
views
Time series convert/summarise series of values into a single value
Hi I have a time series dataset which looks like this
Date Value1
2021/08/01 2
and
...
0
votes
0
answers
11
views
Kernel Density Estimation and performance evaluation
I am doing a data science project about Kernel Density Estimation, specifically about finding the best bandwidth and kernel function to use.
I need to use data that I don't know the actual underlying ...
0
votes
1
answer
20
views
What statistical test should be used for a therapy experiment? [closed]
I'm working on medical project where we propose a new therapy. I have two groups of patients A and B; A follows the usual therapy + the new one and B follows only the usual therapy. the data collected ...
0
votes
1
answer
14
views
Maximum categories for categorical variables in K-Means clustering
I am trying to perform K-means clustering on a dataset, and one of my categorical features has 96 possible options. Would this be too many features for one variable to have? The alternative would be ...
- 101
0
votes
0
answers
14
views
Why do you need more subjects than levels in RM ANOVA?
Honestly, I'm hoping I even asked this question appropriately. It's one of those things I never quite grasped, but just accepted to be true, that when running a within-subjects ANOVA, you have to have ...
- 1
0
votes
0
answers
15
views
Statistical significance on aggregate data to show the groups are difference
I am working with performance data for three groups for each region. The denominator for the groups is the number of people who are identified as low performers. For region A, Group 1 low performer %= ...
- 21
0
votes
0
answers
30
views
Refining AI problem statement - suggestions
I am looking for some guidance. My company is a electronic goods manufacturing company. We work with multiple distributors (around 7 distributors) across specific regions to sell our products. But ...
- 2,449
0
votes
1
answer
17
views
Question about collapsing variable and oversampling minority classes
i have imbalanced data consisting of nine classes, and i am planning to collapse them into two classes. i performed stratified (proportionate) sampling between test, validation, and training sets ...
- 121
0
votes
0
answers
5
views
What are the reasons that can lead to increase Micro F1-score but decrease the Macro F1-score
I have an imbalaced multi-label classification dataset
I tried these 2 models
First I used Bernoulli Naive Bayes algorithm that nativelly supports multi-label classification I got Micro F1-score of 45%...
- 517
0
votes
1
answer
17
views
Combination of groupby and mean methods
I am looking at the below csv file :
We have the question :
Display the mean of the variable gre by group of admitted/not admitted students, using the combination of groupby and mean methods.
I would ...
- 101
0
votes
0
answers
16
views
The meaning of P and degree of freedom in T-Test
I read about T-Test and how we can use it to compare between 2 models (https://towardsdatascience.com/paired-t-test-to-evaluate-machine-learning-classifiers-1f395a6c93fa)
There are some issues I'm not ...
- 363
1
vote
0
answers
36
views
Is there a need to use scaling for Age attribute?
What is the good way working with 'Age' attribute? Don't touch it or should it be scaled?
Below photo shows my results 'Before' and 'After' standardization.
- 111
0
votes
0
answers
9
views
What is the best way to apply Bootstapped weights to data for statistical analysis?
I'm working with a survey dataset for some statistical analysis. My issue is that there are weight columns, but I have no idea how they did the weighting or what the weights are for. I'm not familiar ...
0
votes
0
answers
10
views
Scipy WrappedCauchy isn't wrapping when loc != 0
I want to plot the PDF of a wrapped Cauchy distribution, but when I set a value for the location, the entire distribution gets shifted, rather than wrapped.
When I run this:
...
- 53
0
votes
0
answers
12
views
Robustness of an ARIMA Model
I have built an ARIMAX model in python for predicting a time series. After presenting my findings ive been asked what robustness tests i have used. My skillset is more on the python side. I only have ...
- 131
0
votes
1
answer
34
views
What is the difference between conformal prediction and uncertainty estimation
Recently I am seeing the topic of Conformal Prediction to be very trendy on social media and research. Awesome Conformal Prediction
But what is the main difference between conformal prediction and ...
- 5,776
0
votes
1
answer
10
views
Total Retention Rate Calculated from Categories
I am calculating retention for 3 categories and then total, and I am trying to double check my total, but my check formula isn't working.
I am comparing the last 14 days (let's call it Period 1) to ...
0
votes
0
answers
10
views
Meaning of unit weight for negative impressions in 'Deep Neural Networks for YouTube Recommendations'
I'm having some trouble understanding this section of the paper:
Deep Neural Networks for YouTube Recommendations
4.2 Modeling Expected Watch Time
...
The model is trained with logistic regression ...
- 1
0
votes
0
answers
46
views
Is this function how you would take the Mahalanobis Distance between tensors?
I made an attempt to copy an implementation of the Mahalanobis Distance from the PyTorch library. I'm not sure it is right or if it is more complicated than it needs to be.
I would like a working and ...
- 167
0
votes
0
answers
13
views
why we use chebyshev's inequality when we can convert data into normal distribution?
When there is a normal distribution available, why don't we just converted all data into Russian distribution and then use this equation instead of going with other formula and all we have to buy to ...
0
votes
0
answers
10
views
What is the best way to set up control limits for customer rating data?
I have 2 years of monthly average ratings for a product given by the customers. Obviously there are fluctuations in the average ratings and I wanted to set up limits for this average monthly ratings ...
- 1
0
votes
0
answers
14
views
Is this Time Series Non-Stationary? Or is the ADF test wrong?
I have two time series I want to prove if they are stationary or not. The first time series is the total hours from a YouTube channel over time and the second one is the organic hours (total hours - ...
0
votes
0
answers
7
views
What is the meaning of uncentered variance?
I read some articles that describe ADAM optimizer and they used uncentered variance expression.
I'm not sure I understand the ...
- 363
2
votes
4
answers
157
views
Algorithm to determine whether the first row in CSV is likely to be a header row or a data row
I have a fairly simple problem. I am trying to determine whether the first row in CSV is likely to be a header row or a data row. Looking at single column, the problem can be simplified to: I have a ...
- 121
0
votes
0
answers
20
views
Hypothesis testing
Suppose I am building a machine learning model and I have 20 features. My objective is to understand if there is sufficient statistical evidence of a relationship between the independent features and ...
0
votes
0
answers
19
views
Plotting confidence intervals between points on a graph
I was actually doing some work regarding Bayesian Optimization and came across this graph.
I know this may appear to be a very naive question but.... what is the exact name of this graph (x-f(x) plot) ...
- 35
0
votes
0
answers
23
views
I don't understand why the true error of this classifier is $1/2$
Forgive me. I am very new to machine learning.
Let $S=((x_1,y_1),...,(x_m,y_m))$ be a finite sequence of pairs in $X \times Y$. Assume that the probability distribution $D$ is such that instances are ...
- 1
0
votes
0
answers
11
views
What is the meaning of performance window?
I am looking at this link
https://www.edupristine.com/blog/selecting-the-right-dependent-for-a-scorecard
It says the bad criteria is "30+ days past due" and the performance window is 3 ...
0
votes
0
answers
34
views
Finding one time series signal among 1000 incoming signals
I have one IoT signal, which should be recognized among 1000 incoming IoT signals. However, I receive only values and signalID. What approach would you suggest for this kind of problem?
I have done a ...
- 3
0
votes
0
answers
17
views
Jorda local projections: different size of shocks lead to different conclusions
I use Jorda local projections and I can't get my head around the following.
If I run a Jorda local projection considering a shock of size 1 (say one standard deviation) on my variable of interest X, ...
- 31
0
votes
0
answers
13
views
Why is PACF/ACF graphs not working?
Below I have tried to graph PACF and ACF for my time series data.
However, for some reason the graphs do not properly show the critical blue region that would help me try to choose what p and q values ...
- 11
0
votes
1
answer
41
views
Generate a set of values that has a given correlation with n given other sets of values
Given lists $L_1, \dots, L_n$ of, let's say, 2000 values each and arbitrary numbers $c_1, \dots, c_n$, is it possible to generate a random list of 2000 values that has correlation $c_i$ with $L_i$ for ...
1
vote
1
answer
30
views
g-mean affected by threshold moving?
Is g-mean (geometric mean of sensitivity and specificity) a better evaluation metric than other metrics (that are derived from the confusion matrix) in situations where decision thresholds are not the ...
- 78
0
votes
1
answer
14
views
Whats the best statistical metric for two sets of correlated data?
I have a trained GAN that generates synthetic flight data (9 state variables/features). I plotted the correlation for each pair of state variables similiar to Seaborn's pairplot(). But I have two sets ...
0
votes
1
answer
22
views
How to forecast time bound contract based revenue?
I'm looking at ideas to see how I can forecast contract-based revenue. For example, I can have customers who have purchased a monthly mobile plan, an annual plan, or a 3-year plan. How can I use that ...
0
votes
1
answer
25
views
how to proportionally round to integer values
I am trying to model student progress. Grades are integer values, but progress predictions are fractional.
For example, I have 10 students who are predicted a grade 6, but the model says they should ...
- 3
0
votes
0
answers
13
views
Imputing values for age of property
I have a dataset with property values for a variety of different types of property. I have the age (from build year) of each property however some of the properties are simply "undeveloped ...
0
votes
0
answers
21
views
Determining the information loss due to undersampling
I have an image dataset that I need to segment into directories (train, validation and test) using ImageDataGenerator in TensorFlow/Keras. The dataset is highly imbalanced:
For this I have decided to ...
- 35
0
votes
0
answers
5
views
How to check how similar or different two texts are based on their cluster membership?
I have generated clusters based on crowdsourced ratings of a set of comments (collected from two different platforms). Each comment was rated by three different annotators using a likert scale of 3 ...
- 1