0
$\begingroup$

I currently own a couple of websites and lately I've been implementing some feature changes - I've noticed some changes in website traffic and I was wondering what some of the more sophisticated ways are to measure the significance of the change.

I have 2 scenarios:

  1. I changed one of my website's background from black to orange, this actually increased the site traffic by 17%.

  2. I moved the check out button to a more obvious position on a different site, this specific site unfortunately experienced decreased traffic the day after the implementation by 8%.

I have three questions:

  1. I know the somewhat traditional method of measuring significance, basically run a t-test (look up the z-score in a table and see the probability of this happening by chance, and if its <5% then the change is significant, etc). But is there a more advanced (data-science-y) way of measuring the significance of the change? If it's too much to explain, pointing me to a general direction to look is greatly appreciated!

  2. Speaking of a more data-science-y approach, I've been thinking about triple-exponential smoothing. I can break the day down by hour and build a model of usage by hour, then I can compare the actual hourly usages to the predicted usages and compare the two. Is this a more sophisticated way to go about this or am I just being dumb?

  3. Right now I'm testing out features on different sites in order to minimized interference of the new features I'm implementing. If I implement both these changes to the same site, what are some of the ways I can separate and measure the significance of these two features (kind of similar to partial correlations where I take control of certain variables).

Thanks for reading! Please let me know your thoughts. Any advice would be greatly appreciated.

$\endgroup$
1
$\begingroup$

Welcome to DS stack exchange! Some of your questions/statements are not too clear, I'll try to answer to the best of my abilities but try to ask more precise questions in the future to get better answers (and avoid being down-voted).

To your points 1. and 2.: I would say it's highly unlikely that you'd be able to attribute a small fluctuation like 8% to just moving a button, moreover on such a short time span. And the question you should ask yourself is: "Is this the only thing that you've changed?". The fluctuation in point 2. could easily be attributed to effects like it being the weekend. Two things you'll want to look at to get you started are A/B testing and ARMA models.

Use A/B testing to show some of your customers e.g. a button on a new position or a new background color. Now check how likely they are to return vs. the batch of customers who saw the old design. If the new one is significantly better, then you can be confident that you should roll out the new design.

Set up an ARMA model to predict daily aggregate traffic. I'd advice against smoothing of any kind, it would likely be a bad idea. You first need to set a baseline which you do by comparing e.g. a month's worth of predictions with the actual counts. From this you can estimate the probability distribution of the errors you'd expect from your model. Now when you roll out new features/contents, you can see if the predicted traffic (which is based on data for the old website) is significantly lower that the actual traffic. If so, your new feature/content is a success.

Hope this helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.