0
$\begingroup$

Context

I am working with product data for a retail company. I have the daily impressions (number of times it was viewed online) for all products over a 30 day period (can get more data).

Here is the data for one product:

on_sale = [False, False, False, False, False, False, False, True, True, True, 
True, True, True, True, False, False, False, False, False, False,
False, False, False, False, False, False, False, False, False, False]

impressions = [1,0,0,5,1,0,6,16,17,18,16,17,15,12,4,3,5,4,2,3,2,1,2,4,2,0,3,5,13,3]

So this product was on sale from index 7 to index 13.

enter image description here

We expect that when a product is on sale, it will see an increased number of impressions.

We have introduced a new piece of logic that identifies all on sale products and moves them up into a 'better' advertising campaign, with the aim of increasing their impressions over and above what the normal level of impressions for that on sale product would be.

Question

What is the best way to test the impact of this new piece of logic? In particular, how can I test the impact of moving on sale products into a better campaign (in terms of impressions) relative to what we'd expect for the same on sale products without moving them up a campaign?

I imagine some sort of A/B test where I randomly take some of the on sale products and apply the new logic to them (i.e. move the products up) and compare the change in impressions to a control group of on sale products that I don't apply the new logic to.

Any insight into this problem would be great. I am looking for test that can be implemented in Python.

$\endgroup$

1 Answer 1

0
$\begingroup$

The best way to do a comparison like this is multi-facetted.

The first component is to create an appropriate experiment which means being as completely random as you possibly can and serving an equal number of enhanced and typical advertisement campaigns.

Be super sure that you are being even and fair across times of day, demographics and all possible fronts to limit bias as much as possible.

Ideally I would prefer to have data from before and after the sales collected in both methods as well to give you a sense of how the enhancement works in both sale and non-sale periods of time.

Once you have two sets of data, you need to prepare each set of experiments as a separate time series. You can all the things one might do with time series to explore them separately.

But to do a comparison you can use a granger causality test (here in python, here in R). This test is a test of how well one time series predicts another. It looks at how well a point in set one at a given time is predictive of a value in set two at some time in the future ( some number of lag time increments in the future).

The links explain how it work pretty well and it uses an F-statistic and a p-value like many a good statistical tests.

The underlying idea in using two sets of samples drawn during the same periods with just a change in one dimension implies that a change in pattern is the result of the change in campaign method.

I would use a test on data prior to the sale to see that you have created an experimental sampling method that is sufficiently random. If set one is predictive of set two presale, then you know that a change during the sale which makes set one not predictive of set two should mean that you are no longer sampling from the same population (the new campaign methods is having an affect).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.