Questions tagged [statistics]
Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose.
1,137
questions
0
votes
0
answers
18
views
Interpreting the variance of parameter estimates in linear regression
I am reading through ESL and came across this equation (3.6) where the variance of the parameter estimates are provided as $$Var(\hat{\beta}) = (X^TX)^{-1}{\sigma}^2$$
I can understand the ...
0
votes
0
answers
4
views
Disaggregation algorithm that maintains mass balance?
I am working with monthly timeseries data that I need to represent as daily data. I am looking for a disaggregation algorithm or Python/R package where:
Daily values smoothly increase from one month ...
- 101
1
vote
1
answer
20
views
Correlation of groups together or separately?
Apologies. I am trying to format the table but I am unable to get it in the right format.
I have a tabular time-series data that I am working with and have questions on how to calculate correlation.
I ...
- 11
0
votes
0
answers
6
views
Tackling an automatic labeling (optimization) + classification problem
I'm aware this isn't quite programming-focused, but I did not know any other place to ask this question. It's more about approach than a technical issue.
Context
I have an optimization + ...
- 101
0
votes
0
answers
25
views
How to understand if a model-algorithm is a machine learning ones or not?
I'm working on thesis to detect change points in a timeseries made of body movements. Im forced to not use any Machine Learning models because my colleague used ML and the professor wants to have a ...
- 1
0
votes
0
answers
8
views
statistical test of cell type proportions in clusters
I have a compostion dataframe which contains spotwise compositons of cell types and their cluster informations. I am required to compare the mean abundance of cell types in each cluster with other ...
0
votes
1
answer
20
views
100% Testing Accuracy on Small Image Testing Set
I'm a little worried because I'm getting 100% accuracy on my test set. The test set is relatively small (16 images) because the dataset overall is very small (40 images), and I assume that contributes....
0
votes
0
answers
5
views
Trying to understand the IBM Models for machine translation
I came across some old literature on machine translation, specifically IBM Model 1 here: https://aclanthology.org/J93-2003.pdf. In it, the author makes the makes the statement : "If we replace $\...
- 101
0
votes
0
answers
8
views
Diffusion Model consistency term derivation question
The consistency term of the diffusion model is written as:
$$\mathop{\mathbb{E_{q_\phi(x_{1:T}|x_0)}}} \left[\log\prod_{t=2}^T \frac{p(x_{t-1} | x_t)}{q_\phi(x_{t-1}|x_t, x_0)}\right]$$
$$= \sum_{t=2}^...
0
votes
0
answers
6
views
Within-subject analysis of the impact of energetic music on video game performance: Wilcoxon or Friedman?
I am conducting an amateur study of the impact of music on video game performance across multiple rounds of gameplay.
The gameplay loop is simple: Arrows are falling down, and you have to hit them at ...
- 1
0
votes
0
answers
8
views
What's a suitable error metric for quantifying uncertainty in future projections using a model ensemble?
Rather than just providing a mean projection, I'm looking to provide a likely range of projections using output from 9 models.
Each dataset consists of spatial maximum probability values [0,1] for a ...
- 101
0
votes
0
answers
23
views
When can a low r-squared generate a good predictive models?
Most discussions on model prediction says that you should focus on error metrics, like RMSE, MSE, MAE or MAPE. Some even argue that r-squared can be low in a good model. However, I can't think of a ...
- 61
1
vote
0
answers
6
views
Encouraging sparsity at block level or element-wise level?
I have an objective function $f(W)$, where $W$ is a $Kp \times Kp$ matrix. We can view $W$ is a $p \times p$ block matrix, where each block has the dimension $K \times K$. Now to optimize $f(W)$, I ...
- 11
1
vote
1
answer
48
views
Is it appropriate to use KL Divergence as a loss function for a 1x3 regression model?
I have a regression model with a 1x3 output, which means it predicts three continuous values. I'm wondering if it would be appropriate to use the Kullback-Leibler (KL) Divergence as the loss function ...
- 175
6
votes
2
answers
687
views
How do regression loss functions like MAE and MSE work although they remove the plus/minus sign?
I have a question about regression loss functions like Mean Absolute Error (MAE) and Mean Squared Error (MSE) used in deep learning.
When we calculate these losses, we remove the plus/minus sign from ...
- 175
0
votes
0
answers
11
views
proof of the properties of the whitening transformation
I am looking for a proof that the covariance matrix of a dataset is indeed the unit matrix after the transformation - I could not derive it myself.
Every article I skimmed just had a rough outline ...
- 23
0
votes
1
answer
85
views
Gaussian Process not fitting well // nearly constant predictions
I am a PhD student trying to optimize a chemical engineering process with Bayesian Optimization. I have 5 variables and 3 objectives/responses I want to optimize, so I chose Botorch with the qneHVi ...
- 1
0
votes
0
answers
14
views
Are these statistical test inappropriate?
I have a dataset with the ages of survey respondents and many answers to the survey. Trying to see if there is a relationship between age and survey response. If one was to do individual parametric or ...
- 1
1
vote
0
answers
33
views
Linear Discriminant Analysis (LDA) determining the class for test data after transforming with eigenvectors
Suppose we are given two classes class-1 and class-2 and the mean of the two classes are $μ_1$ and $μ_2$ respectively before projection. Both of their variance-covariance matrix is also provided ...
0
votes
1
answer
39
views
Degree of freedom. Two sampled test
According to this blog/article:
To calculate degrees of freedom for a 2-sample t-test, use N – 2 because there are now two parameters to estimate.
What parameters are they talking about? 2 means: ...
0
votes
0
answers
13
views
How to know the tendencies from a dataset of two columns of categorical values
This is a dataset and my task is to find out the answer of the question: Do men tend to prefer Product A more compared to women?
How to approach? If there was numerical values in both columns we could ...
- 101
0
votes
0
answers
17
views
A/B test question - How to test significance for metrics that are not the unit of randomization
We're runnning an AB test on an ecommerce website. The feature being launched is not for the "users" that come to buy products on the website but is rather for "suppliers" who add ...
- 43
0
votes
0
answers
19
views
Outlier Detection in Zero-inflated dataset
I have a feature , more than 85% of its values are 0s. So when i try traditional boxplot method for outlier detection, all the non-zero values are identified as outliers. What modifications can i make ...
0
votes
0
answers
37
views
Forecasting Resource Depletion in a Distributed System
I manage a distributed system where each node contains six interchangeable resource slots, sourced from a diverse pool of resource types. Each type has a finite number of units, which get consumed ...
- 101
0
votes
0
answers
46
views
When is sampling bias acceptable?
Overview: Dataset is small and a bit messy and the task is to classify 5 classes wherein the targets are ordinal.
Feature Engineering and Selection, Model Tuning, etc. did not produce acceptable ...
0
votes
0
answers
25
views
Adaptive Lasso Coefficient Weights
I'm trying to understand how the Adaptive part of Adaptive Lasso works. I understand that theoretically, the weights for zero coefficients are inflated to infinity. But can someone explain this ...
0
votes
0
answers
22
views
Variable Selection and model prediction
In a supervised problem, I used randomForest for variable selection to identify the most important features.
Question: am I required to use a random forest model for subsequent predictions, or can I ...
0
votes
0
answers
9
views
Synthetic Control for intermittent treatment
I'm exploring the application of the synthetic control method (SCM) to analyze the impact of intermittent treatments or interventions on a target outcome. I'm interested in understanding how SCM can ...
0
votes
0
answers
15
views
How to get probability of an outcome from skewed t distribution in R
I am trying to calculate the probability of stock return to be greater than X in next 28 days, using the skewed t-distribution as it fits the best to the ...
0
votes
0
answers
16
views
Can we calculate Bayes Error rate, if we have a simulated data?
I am going through ISL(Python) and in section 2.2.3 ( Page No. 36), the author writes,
"For our simulated data, the Bayes error is 0.133. It is greater than zero, because the classes overlap in ...
0
votes
0
answers
28
views
Calculation of Covariance Matrices for a QDA classifier in Python Numpy
For a school project, I have to design a QDA classifier for 28x28 pixel images of letters in sign language. I have been given 27455 images for training, which have to be flattened to a 784 pixel ...
0
votes
0
answers
22
views
VC dimension of a class H that assigns 1 to at most k points or that assigns 0 to at most k points
Let $X$ be a finite domain and $k$ a number such that $k\le|X|$. Consider the hypothesis class
$$
H:=\{h:|\{x\in X:h(x)=1\}|\le k\text{ or }|\{x\in X:h(x)=0\}|\le k\}.
$$
Find the VC dimension of $H$.
2
votes
1
answer
202
views
Time Series Analysis and Price Elasticity
Introduction:
As of now, I am a fourth year data science student. As of now, I also have my own company where I work parttime (8/12 hours per week) to gain some more experience in the domain. As you ...
- 21
0
votes
0
answers
22
views
Question on theory from original GBM article
I am reading the original gradient boosting machine article and, maybe because my statistics are a bit rusty, have a few questions on one section.
In section ...
- 11
0
votes
0
answers
21
views
Continuous Function from Binned Data with Consistent Integrals
I have binned energy production data (in Wh) in 5-minute intervals. This means that, at the timestamp of t + 5 minutes, the value represents the amount of energy generated in the interval from t to t +...
0
votes
0
answers
30
views
Generating a Sample T-Distribution in Python
I'm new to DataScience. I'm going through the stats module and the ways to create a sample T-distribution.
Using the below line to generate the sample. However, my question is how is the array ...
1
vote
0
answers
9
views
How to calcualte LR_statistic for the below data
I am trying to calculate LR_statistic for two models where my log-likelihood ratio values as follows.
model1.llf = -2326.9 and model2.llf = -4649.1.
When read ...
1
vote
2
answers
47
views
Am I suppose to do any statistical test on results?
I am doing an email campaign. Before sending emails to users I divided my user base into treatment and control groups (50-50). Divided the groups in such a way that no difference in user behaviour. I ...
2
votes
0
answers
35
views
Best practices for handling "NA" when all NA values exist due to being below the limit of detection?
I am working in R, and have a data set which has a few metabolite concentration values as continuous variables. Anywhere that the concentration was too low to be detected it simply says <LOD. This ...
- 21
1
vote
0
answers
36
views
Find out Statistical test for binary data
we are ran an email campaign for users which is called my treatment and there is control group as well. We want to measure the performance using a Metric called A. If he belongs to metric A then we ...
0
votes
0
answers
16
views
Can an OCR model consistently recognize every digit of a long number correctly?
I'm working on OCR on scanned documents and we need to recognize the exact sequence of some printed numbers on it. Imagine you're reading a bank cheque serial number (16 digits) so the system needs to ...
1
vote
0
answers
34
views
Standard Error interpretation help
I performed a standard error on my machine learning model to predict protein structure. The graph Im showing here is a snippet of the actual data and I deleted some irrelevant info. The y axis is the ...
- 11
0
votes
0
answers
34
views
How to Determine the Minimum Value of a Continuous Variable for Predicting Categorical variable using Logistic Regression?
I am using logistic regression to predict df['MortSubiteCardiaque'], which contains 0 and 1, based on my continuous variable df['NTProBNP']. I would like to determine the threshold for df['NTProBNP'], ...
0
votes
0
answers
20
views
pyspark combined standard deviation?
I have a data frame which consists of multiple standard deviations with mean and count of each. my df is as follows
...
- 101
0
votes
0
answers
16
views
Creating a project based "level of effort remaining" visualization for project tracking and management
I am looking to create a tracker to help manage the current and future progress of a large engineering project. The project is made up of many subprojects each with their own tracker. This is an excel ...
- 1
0
votes
0
answers
14
views
How to find the parameters of custom regression in python code?
I want to make a time series regression that has a custom formula lets say:
Yt=Beta+1/(1+(some parameter sigma))Y(t-1)+(1/(1+(some parameter sigma))**2)Y(t-2)+Beta1X(t-1)+Beta2X(t-2)
how to code this ...
- 1
15
votes
2
answers
740
views
Why does data science see class imbalance as a problem for supervised learning when statistics does not?
Why does data science see class imbalance as a problem in supervised learning when statistics says it is not?
Data science seems to seem class imbalance as problematic and needing special techniques ...
- 4,044
0
votes
0
answers
26
views
Looking for a Statistical Modelling Technique for a Credibility Scoring Model
I’m in the process of developing a model that assigns a credibility score to fatigue reports within an organization. Employees can report feeling “tired” an unlimited number of times throughout the ...
- 261
0
votes
1
answer
26
views
Univariate anomaly / outlier detection
I'm facing a problem that seems 'easy,' but I've been struggling with it for a while now in the field of anomaly/outlier detection.
I have a dataset of around 60K data points. Each data point is part ...
- 1
0
votes
1
answer
446
views
Negative Log Likelihood of a Gaussian Model can be negative?
I am confused about the negative log likelihood of a guassian model:
Can the negative likelihood of a gaussian model be negative ?
lets suppose that the variance is going to 0 faster than the MSE ...
- 1