1
$\begingroup$

we are ran an email campaign for users which is called my treatment and there is control group as well. We want to measure the performance using a Metric called A. If he belongs to metric A then we call the user as 1 else 0.

My data for metric A: This is a sample data we have >1000 sample.

Treatment group : [1,0,1,1,0,0,1]
Control group : [1,0,1,1,0,0,1]

I would like to understand which hypothesis test is best one. I am thinking fisher-exact test or chi-square test.

Improve this question
$\endgroup$
1
  • $\begingroup$ if the counts of both 0 and 1 events are large, then it will make little difference which method is chosen (law of large numbers). For small sample Fisher-exact is conservative (underrejects on average). See notes section for defaults which are recommended in some of the literature in statsmodels.org/dev/generated/… Those methods have better coverage on average but overreject in some cases in small samples. $\endgroup$
    – Josef
    Commented Mar 4 at 16:46

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.