Questions tagged [hypothesis-testing]
The hypothesis-testing tag has no usage guidance.
50 questions
0
votes
0
answers
8
views
What statistical sets should be used to compare 2 dichotomous variables?
You have an ML model that outputs 100,000 out of a million observations of a dichotomous variable 0 or 1. You want to see whether the distribution of the outputted 100,000 observations of the model is ...
- 11
1
vote
0
answers
26
views
Which statistical test is best to compare dichotomous variables?
You have an ML model that outputs 100,000 out of a million observations of a dichotomous variable 0 or 1. You want to see whether the distribution of the outputted 100,000 observations of the model is ...
- 11
0
votes
0
answers
15
views
Are these statistical test inappropriate?
I have a dataset with the ages of survey respondents and many answers to the survey. Trying to see if there is a relationship between age and survey response. If one was to do individual parametric or ...
- 1
0
votes
0
answers
25
views
A/B test question - How to test significance for metrics that are not the unit of randomization
We're runnning an AB test on an ecommerce website. The feature being launched is not for the "users" that come to buy products on the website but is rather for "suppliers" who add ...
- 43
1
vote
2
answers
49
views
Am I suppose to do any statistical test on results?
I am doing an email campaign. Before sending emails to users I divided my user base into treatment and control groups (50-50). Divided the groups in such a way that no difference in user behaviour. I ...
1
vote
0
answers
37
views
Find out Statistical test for binary data
we are ran an email campaign for users which is called my treatment and there is control group as well. We want to measure the performance using a Metric called A. If he belongs to metric A then we ...
0
votes
0
answers
16
views
Correlation in forecast error
I am working on a time series project and while calculating the prediction intervals using bootstrapping method I have to fulfill the assumptions of it. It has an assumption of uncorrelated forecast ...
2
votes
1
answer
179
views
Multiple Hypothesis Testing in feature selection process
I am doing feature selection of features which are of binary nature i.e. each feature represents presence or absence of a substructure in a molecule. And I have a target variable of two classes. My ...
- 77
1
vote
0
answers
38
views
Which inferential test (t-test, F-Test orANOVA) is appropriate here?
I have some sample data of customers and I am looking at the impact of consistent advertising (e.g. showing the same message) to see the impact of click through rates per session.
The analysis shows ...
1
vote
1
answer
91
views
Which statistical procedure to use?
I'm new to Hypothesis testing. I can't really think of how to answer this -
Five students take a test one year and another test the next year. Their overall grades(in %) are given below for the test-1 ...
- 197
1
vote
0
answers
28
views
Is there a statistical test for comparing my dataset percentages with published percentages?
I have a clinical dataset which is showing the percentage of A is 57% and B is 35% for where an tumours is initially found. However the literature is showing it is A is 73% and B is 23%.
Is there a ...
- 11
1
vote
0
answers
43
views
Can you do a power analysis to determine the sample size for a virtual species simulation which is modeled using random forests?
I am simulating virtual species on a 100x100 grid (the size for now). Each grid layer represents one environmental variable. The "suitability function" defines the probability of a presence ...
- 111
2
votes
1
answer
32
views
What can we do in order to find customers that follow a specific pre-defined pattern?
Let's say I have a few customers buying product A. How can I find other customers that have the same characteristics, or something close to, these A buyers?
I was thinking about using clustering ...
2
votes
1
answer
98
views
How to present a statistical justification for the choice of models with approximate accuracies?
In an experiment involving the comparison of classification algorithms, how can I assess whether there are statistically significant differences between the analyzed models? For example, the following ...
0
votes
1
answer
51
views
The meaning of P and degree of freedom in T-Test
I read about T-Test and how we can use it to compare between 2 models (https://towardsdatascience.com/paired-t-test-to-evaluate-machine-learning-classifiers-1f395a6c93fa)
There are some issues I'm not ...
- 527
0
votes
1
answer
48
views
How to demonstrate two variables are orthogonal with respect to the output in a 3-D Python dataset?
I have a Python dataset with 300 samples and 3 columns: 2 independent integer variables X,Y and the dependent continuous variable ...
1
vote
1
answer
21
views
Is systematic sampling a good way to fairly assess a baseline against a post experiment results group?
Apologies if this is in the wrong place. I'm a data analyst thats finding work is driving more and more into into marketing experiments and testing and just needed some help.
Lets say I launch a new ...
- 11
0
votes
1
answer
168
views
"Chi-squared approximation may be incorrect" on hypothesis test
I'm doing a hypothesis test to determine if the data follows binomial distribution with p=0.31, and I'm getting a warning when running this:
...
0
votes
1
answer
40
views
Testing the impact of events on time series
Context
I am working with product data for a retail company. I have the daily impressions (number of times it was viewed online) for all products over a 30 day period (can get more data).
Here is the ...
0
votes
2
answers
94
views
How to check the impact of predictive Model
Suppose that I build an email marketing campaign where the goal is to predict which members will make purchases or not and based on that change the marketing email campaigns. After we build the model, ...
- 11
1
vote
2
answers
90
views
Validate Unsupervised Binary Classification
I’m working on a fully unsupervised anomaly detection problem. Since it’s completely unsupervised, I’m having hard times in defining some metrics to kind of validate the results (I run several ...
- 11
1
vote
1
answer
113
views
How to test the confidence for a rule based system?
I have a multi-class dataset and I generated based on it rules. That is, if certain features are seen then it must be a certain class. I chose only rules with precision 1 (with respect to the whole ...
- 131
2
votes
1
answer
139
views
Which method of correlation is appropriate for two paired lists of numbers?
I have a program which produces an image, and I use a metric to understand how accurate that image is. I choose five cases (A, B, C, D, E), and make a list of the accuracy metric for each case:
...
- 21
1
vote
1
answer
58
views
How to test likelihood hypothesis on dataset?
How to test the following hypothesis?
The larger the fare the more likely the customer is to be travailing alone.
Using the data below, how would one be able to test the hypothesis?
...
- 111
0
votes
2
answers
91
views
Evaluating if metric of one group is higher than the metric of another group when group sizes differ significantly
I am working with a dataset that contains data of applicant income, gender, and loan status (whether or not the person was approved for a loan). I've created the following plots from the data. The ...
- 83
1
vote
1
answer
2k
views
T-test against normalised or standardised data gives different results
I am studying the problem to predict popularity of a tweet, and want to test null hypothesis: there is no relationships between favorite_counts and another set of variables, like number of friends of ...
- 175
1
vote
0
answers
68
views
statistical tests for null hypothesis - what if model is non linear?
I am reading the "An Introduction to Statistical Learning" (Gareth James & alii, Springer) as a primer to machine learning.
I am reading the part in linear regressors, and learnt there ...
- 175
1
vote
1
answer
112
views
Why do I get this result with a chi- square test?
I have a question about the chi squared independence test, I'm working on dataset and I'm interested in finding the link between the categories of product and the gender, I plot my contingency table.
...
- 15
1
vote
1
answer
30
views
How to test whether data is clustered wrt. subcategories?
I have a dataset of about 2000 entries, containing two numerical values, one categorical and one sub-categorical label for each entry. The data is from chemistry lab data, but for the purpose of this ...
user126321
1
vote
0
answers
20
views
What kind of model to use to find drivers when data is aggregated and not on user level?
I have a website and have info from Google Analytics. So I can see the following "features":
page url
country
device category (phone, desktop, etc.)
Number of sessions
Number of users: ...
- 11
1
vote
0
answers
128
views
I am not sure if I used the right hypotheses test or made the right conclusions, could you give me a feedback?
I have just finish a basic statistic course and I decided to apply what I learned in a small project using the wine quality dataset. I am not sure if what I did was correct, so if possible, I would ...
- 231
1
vote
1
answer
141
views
Cross-validation split for modelling data with timeseries behavior
Background: I have a dataset that is generated every month (it is similar with card data that contains card demography and transactions every month and new accounts can be added in the middle of data ...
- 73
0
votes
1
answer
66
views
What is the difference between a test with assumption of standard normal distribution and t-test? And when to use each one?
I am studying statistics on my own and I am not understanding when to use a one-sample t-test or a test with assuming a standard normal distribution. As I understand, both are comparing a population ...
- 231
0
votes
1
answer
101
views
How do I conduct an experiment on the new pricing if it's impossible to conduct an A/B test?
We want to introduce a new price list for the customers of our international SaaS company.
Beforehand we want to test this new price list in several countries.
A/B test cannot be conducted here ...
1
vote
1
answer
47
views
Check if distribution per week is the same
I have sales by customer (b2b) and by date. I want to check if the distribution per day inside weeks remains the same from week to week.
Initial dataset
Customer
Date
Sales
Alpha
2019-02-23
527
...
- 111
0
votes
3
answers
2k
views
Unequal length arrays while calculating paired tests
I have two dfs:
df1 and df2
1 3
2 3
2 4
4 6
6 7
7
9
I want to ...
- 143
4
votes
2
answers
173
views
Which statistical test should I use to show significance in favour of a system?
I have a dataset of the following form:
System A Rating
System B Rating
4.5
5
3
4
5
3
etc.
etc.
I have 155 such data points gathered using a survey. Which statistical test should I use to show ...
- 103
0
votes
1
answer
1k
views
Calculating confidence interval for model accuracy in a multi-class classification problem
In the book Applied Predictive Modeling by Max Kuhn and Kjell Johnson, there is an exercise concerning the calculation of a confidence interval for model accuracy. It reads as follows.
...
- 135
2
votes
1
answer
118
views
Why rejection of a true null hypothesis is called type I error?
I’m comparing two confusion matrices:
https://en.wikipedia.org/wiki/Confusion_matrix#Table_of_confusion
https://en.wikipedia.org/wiki/Type_I_and_type_II_errors
The 2nd is rotated, the Decision is on ...
- 31
4
votes
1
answer
1k
views
How do I know that model performance improvement is significant?
Say I am running a Machine Learning model that produces a certain result (say accuracy of 80%). I now change a minor detail in my model (say, in a Deep Learning model, increase the kernel size in one ...
- 141
2
votes
0
answers
97
views
Comparing two multi-class machine learning classifiers using Stuart Maxwell Test
I need to compare 2 multi-class classifiers. So, to assess whether the difference between the two are statistically significant I have taken the following steps:
obtain prediction on test data using ...
- 31
2
votes
0
answers
133
views
How to apply Hypothesis test on time series?
I am working on a project that is related to Wikipedia's revision history. I have around 10 time-series and each one represents how the relative frequency of positive words changes over time.
I need ...
- 21
4
votes
3
answers
762
views
How do you use KS-test in a data science report?
I'm writing a data science report, I want to find an exist distribution to fit the sample. I got a good looking result , but when I use KS-test to test the model, I got a low p-value,1.2e-4, ...
- 51
7
votes
1
answer
423
views
When to use which multiple testing correction?
There are a large number multiple testing p-value correction methods. e.g.:
...
- 304
1
vote
1
answer
309
views
How to deal with missing values in the survey data to perform Paired sample t test
I have a dataset where I have 100 respondents. Each respondent has to give response on service quality of Health care equipment. Is it providing efficient services to the patients?
We have two ...
- 151
1
vote
0
answers
23
views
How to think about design of experiments in the context of more complex machine learning [closed]
I have experience with multifactor DoE, but in the context of optimizing treatment of a single or or a small number of populations. Are there any articles people recommend to help get my head around ...
- 11
1
vote
1
answer
172
views
How can I compare the grammatical complexity between two texts using their sentences dependency length?
This is a continuation to the following thread.
I have two texts, common English texts such as news articles and informative texts versus a technical textbook. I want to compare the grammatical ...
- 113
2
votes
1
answer
324
views
Interpreting the results based on Granger Causality test
I am trying to use Granger Causality test: https://www.statsmodels.org/stable/generated/statsmodels.tsa.stattools.grangercausalitytests.html
to assess whether "positivity score" affects ...
- 197
2
votes
1
answer
202
views
Permutation test on two groups
I am trying to use a permutation test to test my hypothesis.
I want to make sure I am understanding concept of permutation correctly.
I have control and experimental group. Then I combine them and ...
- 479
8
votes
2
answers
2k
views
A/B testing: How to calculate p-value on post test segments?
My question on A/B testing is about doing post test segmentation analysis.
For example:
I run an A/B test on my website to track bounce rate. On the
treatment group, i put a video to explain ...
- 233