Questions tagged [statistics]

Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.

Filter by
Sorted by
Tagged with
0 votes
0 answers
4 views

how do i compute the predictive covariance matrix from the posterior samples?

I have generated with EMCEE some posterior samples from a statistical model whose likelihood is a multivariate gaussian. it's a regression problem. can you explain me how I can use these samples to ...
  • 101
0 votes
0 answers
12 views

How to fit datasets accounting for standard deviation in Python?

I have multiple datasets of measurements with standard deviation and I would like to fit all the data (with a non-linear regression model) accounting for SD or SEM (it is quite similar to let the ...
0 votes
1 answer
21 views

Statistical test for comparing number of clusters in data

I am performing $K$-means clustering on a dataset consisting of $n$ observations and $d$ variables, and I'm trying to determine the optimal number of clusters. Is there a test that can determine the ...
0 votes
0 answers
15 views

Getting probability to finish in the last 3 ones after each game week

I'am working on a dataframe with five differents features : team game_week season cum_points : pre game cumulative number of points final_position Those datas are covering 10 seasons of Premier ...
0 votes
0 answers
20 views

Which statistical test is suitable for analyze the correlation between non-independent categorical variables?

Im working on an EDA on chocolate bar ratings, trying to indentify relationships between variables. In this case, im trying to test if the ingredients of a chocolate bar area associated with its ...
0 votes
1 answer
14 views

How a Random forest "learns" or How loss (objective function value) is propagated back so that a random forest can "Improve"?

Every Blog and Youtube video talks about the same steps: Choose that you have to build N number of tree and do the task 2-5 ...
  • 323
0 votes
0 answers
18 views

Probability of a Maximum in a Time Series Given Past Data

I'm trying to predict the peak power usage of an EV charging station. I would like figure out probability bounds given the peak power throughout the month. Imagine that our EV time series consists of ...
0 votes
0 answers
17 views

Scaling nominal vars K means

I had a discussion recently with a coworker. We are running a K means clustering algorithm. He said that when dummy variables are made into 1s and 0s,these columns must be scaled in a specific way. If ...
  • 1
1 vote
1 answer
85 views

Trying to extrapolate info from a partial data set - statistical inference

I am wondering if my logic is OK here or not. 98% of a group without a device has an event occur 2% of groups with devices have an event occur. Since we know that correlation isn't causation, I can't ...
  • 63
1 vote
2 answers
36 views

For standard deviation's formula, why does division by sample size come before square rooting?

Why is this used to calculate a sample's standard deviation: $s=\sqrt{\frac{1}{N-1}\sum_{i=1}^N(x_i-\bar{x})^2}$ and not something like: $s=\frac{1}{N-1}\sqrt{\sum_{i=1}^N(x_i-\bar{x})^2}$ I ...
0 votes
1 answer
11 views

What's the word for aggregating by two attributes together, rather than separately?

Let's imagine we have this underlying data: ...
  • 111
0 votes
1 answer
12 views

Time series convert/summarise series of values into a single value

Hi I have a time series dataset which looks like this Date Value1 2021/08/01 2 and ...
0 votes
0 answers
11 views

Kernel Density Estimation and performance evaluation

I am doing a data science project about Kernel Density Estimation, specifically about finding the best bandwidth and kernel function to use. I need to use data that I don't know the actual underlying ...
0 votes
1 answer
20 views

What statistical test should be used for a therapy experiment? [closed]

I'm working on medical project where we propose a new therapy. I have two groups of patients A and B; A follows the usual therapy + the new one and B follows only the usual therapy. the data collected ...
0 votes
1 answer
14 views

Maximum categories for categorical variables in K-Means clustering

I am trying to perform K-means clustering on a dataset, and one of my categorical features has 96 possible options. Would this be too many features for one variable to have? The alternative would be ...
  • 101
0 votes
0 answers
14 views

Why do you need more subjects than levels in RM ANOVA?

Honestly, I'm hoping I even asked this question appropriately. It's one of those things I never quite grasped, but just accepted to be true, that when running a within-subjects ANOVA, you have to have ...
0 votes
0 answers
15 views

Statistical significance on aggregate data to show the groups are difference

I am working with performance data for three groups for each region. The denominator for the groups is the number of people who are identified as low performers. For region A, Group 1 low performer %= ...
0 votes
0 answers
30 views

Refining AI problem statement - suggestions

I am looking for some guidance. My company is a electronic goods manufacturing company. We work with multiple distributors (around 7 distributors) across specific regions to sell our products. But ...
  • 2,449
0 votes
1 answer
17 views

Question about collapsing variable and oversampling minority classes

i have imbalanced data consisting of nine classes, and i am planning to collapse them into two classes. i performed stratified (proportionate) sampling between test, validation, and training sets ...
0 votes
0 answers
5 views

What are the reasons that can lead to increase Micro F1-score but decrease the Macro F1-score

I have an imbalaced multi-label classification dataset I tried these 2 models First I used Bernoulli Naive Bayes algorithm that nativelly supports multi-label classification I got Micro F1-score of 45%...
  • 517
0 votes
1 answer
17 views

Combination of groupby and mean methods

I am looking at the below csv file : We have the question : Display the mean of the variable gre by group of admitted/not admitted students, using the combination of groupby and mean methods. I would ...
  • 101
0 votes
0 answers
16 views

The meaning of P and degree of freedom in T-Test

I read about T-Test and how we can use it to compare between 2 models (https://towardsdatascience.com/paired-t-test-to-evaluate-machine-learning-classifiers-1f395a6c93fa) There are some issues I'm not ...
1 vote
0 answers
36 views

Is there a need to use scaling for Age attribute?

What is the good way working with 'Age' attribute? Don't touch it or should it be scaled? Below photo shows my results 'Before' and 'After' standardization.
0 votes
0 answers
9 views

What is the best way to apply Bootstapped weights to data for statistical analysis?

I'm working with a survey dataset for some statistical analysis. My issue is that there are weight columns, but I have no idea how they did the weighting or what the weights are for. I'm not familiar ...
0 votes
0 answers
10 views

Scipy WrappedCauchy isn't wrapping when loc != 0

I want to plot the PDF of a wrapped Cauchy distribution, but when I set a value for the location, the entire distribution gets shifted, rather than wrapped. When I run this: ...
0 votes
0 answers
12 views

Robustness of an ARIMA Model

I have built an ARIMAX model in python for predicting a time series. After presenting my findings ive been asked what robustness tests i have used. My skillset is more on the python side. I only have ...
  • 131
0 votes
1 answer
34 views

What is the difference between conformal prediction and uncertainty estimation

Recently I am seeing the topic of Conformal Prediction to be very trendy on social media and research. Awesome Conformal Prediction But what is the main difference between conformal prediction and ...
0 votes
1 answer
10 views

Total Retention Rate Calculated from Categories

I am calculating retention for 3 categories and then total, and I am trying to double check my total, but my check formula isn't working. I am comparing the last 14 days (let's call it Period 1) to ...
0 votes
0 answers
10 views

Meaning of unit weight for negative impressions in 'Deep Neural Networks for YouTube Recommendations'

I'm having some trouble understanding this section of the paper: Deep Neural Networks for YouTube Recommendations 4.2 Modeling Expected Watch Time ... The model is trained with logistic regression ...
  • 1
0 votes
0 answers
46 views

Is this function how you would take the Mahalanobis Distance between tensors?

I made an attempt to copy an implementation of the Mahalanobis Distance from the PyTorch library. I'm not sure it is right or if it is more complicated than it needs to be. I would like a working and ...
  • 167
0 votes
0 answers
13 views

why we use chebyshev's inequality when we can convert data into normal distribution?

When there is a normal distribution available, why don't we just converted all data into Russian distribution and then use this equation instead of going with other formula and all we have to buy to ...
0 votes
0 answers
10 views

What is the best way to set up control limits for customer rating data?

I have 2 years of monthly average ratings for a product given by the customers. Obviously there are fluctuations in the average ratings and I wanted to set up limits for this average monthly ratings ...
  • 1
0 votes
0 answers
14 views

Is this Time Series Non-Stationary? Or is the ADF test wrong?

I have two time series I want to prove if they are stationary or not. The first time series is the total hours from a YouTube channel over time and the second one is the organic hours (total hours - ...
0 votes
0 answers
7 views

What is the meaning of uncentered variance?

I read some articles that describe ADAM optimizer and they used uncentered variance expression. I'm not sure I understand the ...
2 votes
4 answers
157 views

Algorithm to determine whether the first row in CSV is likely to be a header row or a data row

I have a fairly simple problem. I am trying to determine whether the first row in CSV is likely to be a header row or a data row. Looking at single column, the problem can be simplified to: I have a ...
0 votes
0 answers
20 views

Hypothesis testing

Suppose I am building a machine learning model and I have 20 features. My objective is to understand if there is sufficient statistical evidence of a relationship between the independent features and ...
0 votes
0 answers
19 views

Plotting confidence intervals between points on a graph

I was actually doing some work regarding Bayesian Optimization and came across this graph. I know this may appear to be a very naive question but.... what is the exact name of this graph (x-f(x) plot) ...
0 votes
0 answers
23 views

I don't understand why the true error of this classifier is $1/2$

Forgive me. I am very new to machine learning. Let $S=((x_1,y_1),...,(x_m,y_m))$ be a finite sequence of pairs in $X \times Y$. Assume that the probability distribution $D$ is such that instances are ...
  • 1
0 votes
0 answers
11 views

What is the meaning of performance window?

I am looking at this link https://www.edupristine.com/blog/selecting-the-right-dependent-for-a-scorecard It says the bad criteria is "30+ days past due" and the performance window is 3 ...
0 votes
0 answers
34 views

Finding one time series signal among 1000 incoming signals

I have one IoT signal, which should be recognized among 1000 incoming IoT signals. However, I receive only values and signalID. What approach would you suggest for this kind of problem? I have done a ...
  • 3
0 votes
0 answers
17 views

Jorda local projections: different size of shocks lead to different conclusions

I use Jorda local projections and I can't get my head around the following. If I run a Jorda local projection considering a shock of size 1 (say one standard deviation) on my variable of interest X, ...
  • 31
0 votes
0 answers
13 views

Why is PACF/ACF graphs not working?

Below I have tried to graph PACF and ACF for my time series data. However, for some reason the graphs do not properly show the critical blue region that would help me try to choose what p and q values ...
0 votes
1 answer
41 views

Generate a set of values that has a given correlation with n given other sets of values

Given lists $L_1, \dots, L_n$ of, let's say, 2000 values each and arbitrary numbers $c_1, \dots, c_n$, is it possible to generate a random list of 2000 values that has correlation $c_i$ with $L_i$ for ...
1 vote
1 answer
30 views

g-mean affected by threshold moving?

Is g-mean (geometric mean of sensitivity and specificity) a better evaluation metric than other metrics (that are derived from the confusion matrix) in situations where decision thresholds are not the ...
  • 78
0 votes
1 answer
14 views

Whats the best statistical metric for two sets of correlated data?

I have a trained GAN that generates synthetic flight data (9 state variables/features). I plotted the correlation for each pair of state variables similiar to Seaborn's pairplot(). But I have two sets ...
0 votes
1 answer
22 views

How to forecast time bound contract based revenue?

I'm looking at ideas to see how I can forecast contract-based revenue. For example, I can have customers who have purchased a monthly mobile plan, an annual plan, or a 3-year plan. How can I use that ...
0 votes
1 answer
25 views

how to proportionally round to integer values

I am trying to model student progress. Grades are integer values, but progress predictions are fractional. For example, I have 10 students who are predicted a grade 6, but the model says they should ...
  • 3
0 votes
0 answers
13 views

Imputing values for age of property

I have a dataset with property values for a variety of different types of property. I have the age (from build year) of each property however some of the properties are simply "undeveloped ...
0 votes
0 answers
21 views

Determining the information loss due to undersampling

I have an image dataset that I need to segment into directories (train, validation and test) using ImageDataGenerator in TensorFlow/Keras. The dataset is highly imbalanced: For this I have decided to ...
0 votes
0 answers
5 views

How to check how similar or different two texts are based on their cluster membership?

I have generated clusters based on crowdsourced ratings of a set of comments (collected from two different platforms). Each comment was rated by three different annotators using a likert scale of 3 ...
  • 1

1
2 3 4 5
23