Questions tagged [hypothesis-testing]
The hypothesis-testing tag has no usage guidance.
51
questions
1
vote
0
answers
40
views
what is the genesis of the term - the clinical significance?
The literature suggests that p-value represents "clinical" significance.
(p-value, is a number whose value can range from zero to one.)To me,p-value does not measure clinical significance....
- 599
1
vote
1
answer
63
views
Which statistical procedure to use?
I'm new to Hypothesis testing. I can't really think of how to answer this -
Five students take a test one year and another test the next year. Their overall grades(in %) are below for both courses. ...
- 197
0
votes
0
answers
19
views
AB testing: Control was performing 0.5% better than experiment set before the initiation of experiment
So we introduced a new feature in our app, that would aid conversion (hypothetically).
When i tried to measure this incremental change in conversion, i split my base set of customers into control(C) ...
- 408
0
votes
0
answers
14
views
Testing Marketing vehicle effectiveness
I have a set of customers who have been targetted using a marketing vehicle. Now i want to estimate if the marketing vehicle resulted in a better conversion (did they buy after that or not).
Currently,...
1
vote
0
answers
24
views
Is there a statistical test for comparing my dataset percentages with published percentages?
I have a clinical dataset which is showing the percentage of A is 57% and B is 35% for where an tumours is initially found. However the literature is showing it is A is 73% and B is 23%.
Is there a ...
- 11
0
votes
0
answers
7
views
Feature Selection using Statistical Testing on Features with Different Scales
If I want to select features based on ANOVA test for example, should I scale features to same/similar range so the results can be compared , or is it unnecessary?
- 51
1
vote
0
answers
31
views
Can you do a power analysis to determine the sample size for a virtual species simulation which is modeled using random forests?
I am simulating virtual species on a 100x100 grid (the size for now). Each grid layer represents one environmental variable. The "suitability function" defines the probability of a presence ...
- 111
2
votes
1
answer
26
views
What can we do in order to find customers that follow a specific pre-defined pattern?
Let's say I have a few customers buying product A. How can I find other customers that have the same characteristics, or something close to, these A buyers?
I was thinking about using clustering ...
2
votes
1
answer
88
views
How to present a statistical justification for the choice of models with approximate accuracies?
In an experiment involving the comparison of classification algorithms, how can I assess whether there are statistically significant differences between the analyzed models? For example, the following ...
0
votes
1
answer
32
views
The meaning of P and degree of freedom in T-Test
I read about T-Test and how we can use it to compare between 2 models (https://towardsdatascience.com/paired-t-test-to-evaluate-machine-learning-classifiers-1f395a6c93fa)
There are some issues I'm not ...
- 363
0
votes
0
answers
27
views
Paired T-Test to assess validity of Algorithm Selection after Nested Cross Validation
I am interested in selecting the best Algorithm (SVM, XGBoost, Random Forest, ...) for a classification problem.
At present, my company is using an algorithm (let's call it ALG-A and let's say it's ...
0
votes
0
answers
20
views
A/B test results don't hold in hold-out dataset
I'm working on a project consisting in analyzing some game data. The game consists of two players competing between them. The result of the game is either win or lose. Players can choose from ...
- 536
0
votes
1
answer
23
views
How to demonstrate two variables are orthogonal with respect to the output in a 3-D Python dataset?
I have a Python dataset with 300 samples and 3 columns: 2 independent integer variables X,Y and the dependent continuous variable ...
0
votes
0
answers
30
views
Why p-value larger than level of significant should accept the hypothesis?
I hope someone explains this question to me like I'm ten. I already know why (sample mean - population mean) < critical value should accept the hypothesis.
- 13
0
votes
0
answers
32
views
Kolmogorov-Smirnov hypothesis test
I am using this package to test the similarity of two distributions.
It says:
The one-sample test compares the underlying distribution F(x) of a
sample against a given distribution G(x). The two-...
- 105
0
votes
0
answers
11
views
hypothesis testing for difference between groups of embedding vectors
In a machine learning application, I have two generators G₁ and G₂ that generate descriptions of some input z in the form of embedding vectors X and Y:
Xᵢ = [x₁, x₂, ...] = G₁(zᵢ)
Yⱼ = [y₁, y₂, ...] =...
- 1
1
vote
1
answer
17
views
Is systematic sampling a good way to fairly assess a baseline against a post experiment results group?
Apologies if this is in the wrong place. I'm a data analyst thats finding work is driving more and more into into marketing experiments and testing and just needed some help.
Lets say I launch a new ...
- 11
0
votes
1
answer
68
views
"Chi-squared approximation may be incorrect" on hypothesis test
I'm doing a hypothesis test to determine if the data follows binomial distribution with p=0.31, and I'm getting a warning when running this:
...
0
votes
1
answer
34
views
Testing the impact of events on time series
Context
I am working with product data for a retail company. I have the daily impressions (number of times it was viewed online) for all products over a 30 day period (can get more data).
Here is the ...
0
votes
2
answers
86
views
How to check the impact of predictive Model
Suppose that I build an email marketing campaign where the goal is to predict which members will make purchases or not and based on that change the marketing email campaigns. After we build the model, ...
- 11
1
vote
2
answers
50
views
Validate Unsupervised Binary Classification
I’m working on a fully unsupervised anomaly detection problem. Since it’s completely unsupervised, I’m having hard times in defining some metrics to kind of validate the results (I run several ...
- 11
1
vote
1
answer
65
views
How to test the confidence for a rule based system?
I have a multi-class dataset and I generated based on it rules. That is, if certain features are seen then it must be a certain class. I chose only rules with precision 1 (with respect to the whole ...
- 131
2
votes
2
answers
49
views
Which method of correlation is appropriate for two paired lists of numbers?
I have a program which produces an image, and I use a metric to understand how accurate that image is. I choose five cases (A, B, C, D, E), and make a list of the accuracy metric for each case:
...
- 21
1
vote
1
answer
48
views
How to test likelihood hypothesis on dataset?
How to test the following hypothesis?
The larger the fare the more likely the customer is to be travailing alone.
Using the data below, how would one be able to test the hypothesis?
...
- 111
0
votes
2
answers
86
views
Evaluating if metric of one group is higher than the metric of another group when group sizes differ significantly
I am working with a dataset that contains data of applicant income, gender, and loan status (whether or not the person was approved for a loan). I've created the following plots from the data. The ...
- 73
1
vote
1
answer
1k
views
T-test against normalised or standardised data gives different results
I am studying the problem to predict popularity of a tweet, and want to test null hypothesis: there is no relationships between favorite_counts and another set of variables, like number of friends of ...
- 165
1
vote
0
answers
62
views
statistical tests for null hypothesis - what if model is non linear?
I am reading the "An Introduction to Statistical Learning" (Gareth James & alii, Springer) as a primer to machine learning.
I am reading the part in linear regressors, and learnt there ...
- 165
1
vote
1
answer
92
views
Why do I get this result with a chi- square test?
I have a question about the chi squared independence test, I'm working on dataset and I'm interested in finding the link between the categories of product and the gender, I plot my contingency table.
...
- 15
1
vote
1
answer
26
views
How to test whether data is clustered wrt. subcategories?
I have a dataset of about 2000 entries, containing two numerical values, one categorical and one sub-categorical label for each entry. The data is from chemistry lab data, but for the purpose of this ...
user126321
1
vote
0
answers
18
views
What kind of model to use to find drivers when data is aggregated and not on user level?
I have a website and have info from Google Analytics. So I can see the following "features":
page url
country
device category (phone, desktop, etc.)
Number of sessions
Number of users: ...
- 11
1
vote
0
answers
104
views
I am not sure if I used the right hypotheses test or made the right conclusions, could you give me a feedback?
I have just finish a basic statistic course and I decided to apply what I learned in a small project using the wine quality dataset. I am not sure if what I did was correct, so if possible, I would ...
- 223
1
vote
1
answer
131
views
Cross-validation split for modelling data with timeseries behavior
Background: I have a dataset that is generated every month (it is similar with card data that contains card demography and transactions every month and new accounts can be added in the middle of data ...
- 73
1
vote
1
answer
46
views
What is the difference between a hypothesis test made with a normal distribution and a one sample t-test? And when to use each one?
I am studying statistics on my own and I am not understanding when to use a one sample t-test or a test with a normal distribution. As I understand, both are comparing a population mean with a sample ...
- 223
0
votes
1
answer
72
views
How do I conduct an experiment on the new pricing if it's impossible to conduct an A/B test?
We want to introduce a new price list for the customers of our international SaaS company.
Beforehand we want to test this new price list in several countries.
A/B test cannot be conducted here ...
1
vote
1
answer
43
views
Check if distribution per week is the same
I have sales by customer (b2b) and by date. I want to check if the distribution per day inside weeks remains the same from week to week.
Initial dataset
Customer
Date
Sales
Alpha
2019-02-23
527
...
- 11
0
votes
3
answers
1k
views
Unequal length arrays while calculating paired tests
I have two dfs:
df1 and df2
1 3
2 3
2 4
4 6
6 7
7
9
I want to ...
- 143
4
votes
2
answers
149
views
Which statistical test should I use to show significance in favour of a system?
I have a dataset of the following form:
System A Rating
System B Rating
4.5
5
3
4
5
3
etc.
etc.
I have 155 such data points gathered using a survey. Which statistical test should I use to show ...
- 103
0
votes
1
answer
843
views
Calculating confidence interval for model accuracy in a multi-class classification problem
In the book Applied Predictive Modeling by Max Kuhn and Kjell Johnson, there is an exercise concerning the calculation of a confidence interval for model accuracy. It reads as follows.
...
- 115
2
votes
1
answer
93
views
Why rejection of a true null hypothesis is called type I error?
I’m comparing two confusion matrices:
https://en.wikipedia.org/wiki/Confusion_matrix#Table_of_confusion
https://en.wikipedia.org/wiki/Type_I_and_type_II_errors
The 2nd is rotated, the Decision is on ...
- 21
4
votes
2
answers
490
views
How do I know that model performance improvement is significant?
Say I am running a Machine Learning model that produces a certain result (say accuracy of 80%). I now change a minor detail in my model (say, in a Deep Learning model, increase the kernel size in one ...
- 141
2
votes
0
answers
47
views
Comparing two multi-class machine learning classifiers using Stuart Maxwell Test
I need to compare 2 multi-class classifiers. So, to assess whether the difference between the two are statistically significant I have taken the following steps:
obtain prediction on test data using ...
- 31
2
votes
0
answers
116
views
How to apply Hypothesis test on time series?
I am working on a project that is related to Wikipedia's revision history. I have around 10 time-series and each one represents how the relative frequency of positive words changes over time.
I need ...
- 21
4
votes
2
answers
503
views
How do you use KS-test in a data science report?
I'm writing a data science report, I want to find an exist distribution to fit the sample. I got a good looking result , but when I use KS-test to test the model, I got a low p-value,1.2e-4, ...
- 51
7
votes
1
answer
317
views
When to use which multiple testing correction?
There are a large number multiple testing p-value correction methods. e.g.:
...
- 284
1
vote
1
answer
73
views
How to deal with missing values in the survey data to perform Paired sample t test
I have a dataset where I have 100 respondents. Each respondent has to give response on service quality of Health care equipment. Is it providing efficient services to the patients?
We have two ...
- 151
1
vote
0
answers
23
views
How to think about design of experiments in the context of more complex machine learning [closed]
I have experience with multifactor DoE, but in the context of optimizing treatment of a single or or a small number of populations. Are there any articles people recommend to help get my head around ...
- 11
0
votes
0
answers
38
views
Hypothesis Testing - Median Test for linear regression model given standard error and co-efficient
I need help approaching this particular problem on hypothesis testing.
A researcher believes that a diet high in raw cornstarch would increase the median blood glucose levels by 20. The diet can ...
1
vote
1
answer
91
views
How can I compare the grammatical complexity between two texts using their sentences dependency length?
This is a continuation to the following thread.
I have two texts, common English texts such as news articles and informative texts versus a technical textbook. I want to compare the grammatical ...
- 113
2
votes
1
answer
250
views
Interpreting the results based on Granger Causality test
I am trying to use Granger Causality test: https://www.statsmodels.org/stable/generated/statsmodels.tsa.stattools.grangercausalitytests.html
to assess whether "positivity score" affects ...
- 197
2
votes
1
answer
109
views
Permutation test on two groups
I am trying to use a permutation test to test my hypothesis.
I want to make sure I am understanding concept of permutation correctly.
I have control and experimental group. Then I combine them and ...
- 385