I am doing an email campaign. Before sending emails to users I divided my user base into treatment and control groups (50-50). Divided the groups in such a way that no difference in user behaviour. I sent email only to treatment group.

I have some other metrics to measure for success of campaign apart from CTR, Open rate. I have measured different business metrics for the 7 days prior & post of email send date for both control and treatment groups. I got a uplift of 10% for treatment group and 4.5% for control group.

My question here is: am I supposed to conduct a statistical significance test on the results of treatment and control groups? or can I say directly that treatment group is doing better than control because 10% > 4.5%?

Note : It is not an assignment kind of project. I am doing a email recommendation project.

  • $\begingroup$ please "clarify have the results of those for both groups" and what do you want to say - "sample recommendation" $\endgroup$ Commented Mar 23 at 22:38
  • $\begingroup$ @SubhashC.Davar- I have edited please check it. $\endgroup$ Commented May 26 at 10:23

1 Answer 1


You can't say directly that treatment is doing better than control because 10% > 4.5%. Further, you can administer a statistical significance test to ascertain whether the difference in two proportions is due to chance i.e. sampling fluctuations at a specified alpha level. Here, you may apply t-test.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.