1
$\begingroup$

Apologies if this is in the wrong place. I'm a data analyst thats finding work is driving more and more into into marketing experiments and testing and just needed some help.

Lets say I launch a new feature on my site that I think will bring in new core users of my product. I keep keep this feature live for two weeks and track users who sign-up because of it. And the end of the two weeks I end up with say 140 new users who signed up because of the feature.

I now want to assess if these new users I obtained were more engaged with the product than other users who signed up across the two week time frame. Lets say I had 2520 total users signup in that two week period, giving 140 experiment sign-up users and 17 more random groups of 140. I then assess if the experiment group had a higher or lower engagement rate than the other 17 random groups. Is this a fair and solid way to assess if the the new feature had true significant effect on user engagement outside of chance?

$\endgroup$
1
  • $\begingroup$ Also, I think Cross Validated is more suitable for this type of questions. $\endgroup$
    – Valentas
    Oct 21, 2022 at 13:15

1 Answer 1

1
$\begingroup$

Yes, I think this experiment could provide you some interesting information.

However, in the case you see a significant difference, you will still have the correlation $\ne$ causation problem (it can be that some users used the feature because they were more engaged from the start, and not that your feature caused them to be more engaged). To avoid this problem you need to do A/B testing.

A side note: for simple bootstrap you could alternatively sample many groups of 140 with replacement, rather than consider fixed 17 groups.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.