I currently own a couple of websites and lately I've been implementing some feature changes - I've noticed some changes in website traffic and I was wondering what some of the more sophisticated ways are to measure the significance of the change.
I have 2 scenarios:
I changed one of my website's background from black to orange, this actually increased the site traffic by 17%.
I moved the check out button to a more obvious position on a different site, this specific site unfortunately experienced decreased traffic the day after the implementation by 8%.
I have three questions:
I know the somewhat traditional method of measuring significance, basically run a t-test (look up the z-score in a table and see the probability of this happening by chance, and if its <5% then the change is significant, etc). But is there a more advanced (data-science-y) way of measuring the significance of the change? If it's too much to explain, pointing me to a general direction to look is greatly appreciated!
Speaking of a more data-science-y approach, I've been thinking about triple-exponential smoothing. I can break the day down by hour and build a model of usage by hour, then I can compare the actual hourly usages to the predicted usages and compare the two. Is this a more sophisticated way to go about this or am I just being dumb?
Right now I'm testing out features on different sites in order to minimized interference of the new features I'm implementing. If I implement both these changes to the same site, what are some of the ways I can separate and measure the significance of these two features (kind of similar to partial correlations where I take control of certain variables).
Thanks for reading! Please let me know your thoughts. Any advice would be greatly appreciated.