Am I suppose to do any statistical test on results?

Question

I am doing an email campaign. Before sending emails to users I divided my user base into treatment and control groups (50-50). Divided the groups in such a way that no difference in user behaviour. I sent email only to treatment group.

I have some other metrics to measure for success of campaign apart from CTR, Open rate. I have measured different business metrics for the 7 days prior & post of email send date for both control and treatment groups. I got a uplift of 10% for treatment group and 4.5% for control group.

My question here is: am I supposed to conduct a statistical significance test on the results of treatment and control groups? or can I say directly that treatment group is doing better than control because 10% > 4.5%?

Note : It is not an assignment kind of project. I am doing a email recommendation project.

Answer 1

You can't say directly that treatment is doing better than control because 10% > 4.5%. Further, you can administer a statistical significance test to ascertain whether the difference in two proportions is due to chance i.e. sampling fluctuations at a specified alpha level. Here, you may apply t-test.

Answer 2

Yes I believe you should conduct a statistical significance test on the results of the treatment and control groups, even though the treatment group shows a higher uplift (10% vs. 4.5%).