Apologies if this is in the wrong place. I'm a data analyst thats finding work is driving more and more into into marketing experiments and testing and just needed some help.
Lets say I launch a new feature on my site that I think will bring in new core users of my product. I keep keep this feature live for two weeks and track users who sign-up because of it. And the end of the two weeks I end up with say 140 new users who signed up because of the feature.
I now want to assess if these new users I obtained were more engaged with the product than other users who signed up across the two week time frame. Lets say I had 2520 total users signup in that two week period, giving 140 experiment sign-up users and 17 more random groups of 140. I then assess if the experiment group had a higher or lower engagement rate than the other 17 random groups. Is this a fair and solid way to assess if the the new feature had true significant effect on user engagement outside of chance?